https://orcid.org/0000-0002-3956-5223
Figures
Abstract
Background
Recent technological and radiological advances have renewed interest in using X-rays to screen and triage people with tuberculosis (TB). The miniaturization of digital X-ray (DXR), combined with automatic interpretation using computer-aided detection (CAD) software can extend the reach of DXR screening interventions for TB. This qualitative study assessed early implementers’ experiences and lessons learned when using ultra-portable (UP) DXR systems integrated with CAD software to screen and triage TB.
Methods
Semi-structured interviews were conducted with project staff and healthcare workers at six pilot sites. Transcripts were coded and analyzed using a framework approach. The themes that emerged were subsequently organized and presented using the Consolidated Framework for Implementation Research (CFIR).
Results
There were 26 interviewees with varying roles: supervisory, clinicians, radiographers, and radiologists. Participants recognized the portability as the main advantage, but criticize that it involves several compromises on throughput, internet dependence, manoeuvrability, and stability, as well as suitability for patients with larger body sizes. Furthermore, compared to using hardware and software from the same supplier and without digital health information systems, complexity increases with interoperability between hardware and software, and between different electronic health information systems. Currently, there is a limited capacity to implement these technologies, especially due to the need for threshold selection, and lack of guidance on radiation protection suitable for UP DXR machines. Finally, the respondents stressed the importance of having protected means of sharing patient medical data, as well as comprehensive support and warranty plans.
Conclusion
Study findings suggest that UP DXR with CAD was overall well received to decentralize radiological assessment for TB, however, the improved portability involved programmatic compromises. The main barriers to uptake included insufficient capacity and lack of guidance on radiation protection suitable for UP DXR.
Citation: Qin ZZ, Barrett R, del Mar Castro M, Zaidi S, Codlin AJ, Creswell J, et al. (2023) Early user experience and lessons learned using ultra-portable digital X-ray with computer-aided detection (DXR-CAD) products: A qualitative study from the perspective of healthcare providers. PLoS ONE 18(2): e0277843. https://doi.org/10.1371/journal.pone.0277843
Editor: Sathishkumar V. E., Hanyang University, REPUBLIC OF KOREA
Received: November 2, 2022; Accepted: February 9, 2023; Published: February 24, 2023
Copyright: © 2023 Qin et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Our dataset is available at: https://github.com/ZZQin/UPXR-AI_Qualitative. The personal information of the participants and all other confidential information has been stored and secured according to the medical confidentiality and other provisions of the General Data Protection Regulation (GDPR), the German Federal Data Protection Act (BDSG) and the associated Baden-Württemberg State Data Protection Act (LDSG BW).
Funding: Global Affairs Canada funded the study through TB REACH for authors ZZQ and JC The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
1 Introduction
Tuberculosis (TB) is a transmissible, bacteriological disease behind a global health crisis. TB kills more people every year than any other infectious disease despite the availability of effective treatments and was responsible for 1.6 million deaths worldwide in 2021, according to the World Health Organization (WHO) [1]. The global TB burden is concentrated in lower middle-income countries, eight of which account for two thirds of incident TB [2]. While COVID-19 has recently set back the international campaign to eliminate TB by at least five years, even before the COVID-19 pandemic, the global campaign against TB was not on track to meet the targets set by the Global Plan to End TB [3].
Of the estimated 10.6 million people who fell ill with TB in 2021, around 4.2 million were not diagnosed [4]. This persistent gap between TB incidence and notification has driven renewed interest in systematic screening for TB [4]. This change has taken place against a backdrop of a fundamental paradigm shift from TB control towards TB elimination and significant technological advances [5]. The most traditional method of screening relies on symptoms, which has low sensitivity considering that, on average, 50.4% of confirmed TB cases are asymptomatic [4,6]. In contrast, chest X-ray (CXR) is a highly sensitive screening method able to support identification of asymptomatic cases [4,6].
New detector panel technology has spurred innovation in digital X-ray (DXR) technology to produce smaller, battery-powered devices with reduced radiation emissions. These are designed to be portable in a suitcase, backpack, or even hand-held. These ultra-portable devices allows CXR screening to take place in remote, hard-to-reach locations compared to traditional, heavy-duty stationary devices that are concentrated in health facilities. Limited availability of radiologists in lower-income settings pose another challenge to TB screening [7]. Computer-aided detection (CAD) software can provide automatic and standardized interpretation of CXR in the absence of radiologists, which potentially circumvents issues related to CXR screening, such as inter- and intra-reader variability and the lack of skilled radiologists in high-burden countries [4,7–9]. A recent landscape review identified 13 CAD products that are commercially available for TB detection, with new products and product versions in the pipeline [10]. Independent evaluations show CAD software can read CXR just as accurately as, or even better than, human readers [8,9]; leading to the recommendation of CAD by WHO in the 2021 update to its TB screening guidelines [4]. Applying ultra-portable DXR with CAD software (UP DXR-CAD) offers an opportunity to address the barriers associated with CXR screening in the past, thereby improving TB detection and initiation of treatment.
To date, the literature on using CAD for TB screening has focused primarily on performance evaluation and the accuracy of different CAD software [8,9,11,12]. Meanwhile, the experiences of early CAD implementers, and the barriers and facilitators they face in TB screening projects, have not been explored previously, leaving an gap for implementation and planning by national programs. Important questions also remain around the potential of UP-DXR technology, which has also not been well documented in literature to-date. Understanding user experiences is key to guide the scale-up of this technology and achieve its adoption by national TB programs. Thus, failing to incorporate users’ experiences and perspectives in the implementation of CXR CADw would lead to limited or slow uptake, thus delaying or impeding reaching the full potential of this novel technology for TB control. Seeking to address this gap, we conducted a qualitative study with early implementers of CAD for DXR TB screening. To our knowledge, this is the first study to use qualitative methods to explore experiences of users of this new technology along the constructs of the Consolidated Framework for Implementation Research (CFIR). Our objective was to uncover the acceptability, lessons learned and perceived limitations of the technology from the health provider perspective, to guide programmatic implementation.
2 Methods
2.1 Ethics statement
This research was approved and monitored by Ethikkommission der Medizinischen Fakultät Heidelberg (reference number: S-399/2021) in accordance with consensus ethical principles derived from international guidelines, including the Declaration of Helsinki (7th revision, 2013). The participants, all above 18 years of age, were informed that their participation in the study was voluntary, and that they could submit written or oral notification of withdrawal at any time, before or after the interview, without explanation or penalty. Written informed consent was obtained from all participants, with some wishing to be acknowledged as contributors and some remaining anonymous. The personal information of the participants and all other confidential information has been stored and secured according to the medical confidentiality and other provisions of the General Data Protection Regulation (GDPR), the German Federal Data Protection Act (BDSG) and the associated Baden-Württemberg State Data Protection Act (LDSG BW).
2.2 Context and participants
UP DXR-CAD can be integrated into TB screening programmes in various ways. This includes parallel and sequential with symptom screening. In a parallel scenario, everyone undergoes CXR and symptom screening and those with either symptoms suggestive of TB or an abnormal CXR according to a pre-set CAD threshold score is referred for confirmatory bacteriological testing (Fig 1A). Alternatively, in a sequential screening scenario, people identified with symptoms indicative of TB are then screened with UP DXR-CAD and those who have an abnormal CXR are referred for bacteriological testing (Fig 1B). The location of the screening may vary from community-based in hard-to-reach areas to hospital based (Table 1).
a: Parallel screening algorithm. b: Sequential screening algorithm.
At the time of the interviews, all implementation of DXR-CAD had been conducted in small-scale pilot settings; therefore, none of the interviews correspond to a programmatic setting. Participants were identified among staff of projects known to be using DXR-CAD for TB screening. This was achieved through the network of in-country partners of the Stop TB Partnership’s TB REACH initiative in several high TB burden countries, including Cambodia, Nigeria, Pakistan, Uganda, Vietnam, and Zambia.
Six DXR-CAD pilot projects were identified. The projects in Nigeria and Vietnam deployed DXR-CAD to support community-based active TB case finding (ACF) in hard-to-reach areas, such as remote or mountainous areas and islands. The remaining four projects primarily employed facility-based screening; in Zambia and Cambodia DXR-CAD was used in remote clinics, while in Pakistan these tools were used in tertiary hospitals, and in Ghana they were used for systematic screening in prisons. In both Cambodia and Pakistan, these tools were also used for community-based ACF in rural and mining areas, similar to the projects in Nigeria and Vietnam. Four ultra-portable DXR systems (Fujifilm FDR Xair, Delft Ultra, Delft Light, MINE2) and three CAD software products (Lunit INSIGHT CXR, qXR, CAD4TB) were used by the six projects in a range of combinations (Table 1). All the equipment was commercially available at the time of implementation.
We interviewed representatives from the six projects following a purposive sampling approach to identify information-rich case studies that best reveal user experiences with DXR-CAD. To guide recruitment, we followed the concept of information power, whereby a targeted group of participants were recruited as they met the specific aims of gathering quality information from both health workers and programmatic personnel involved in the early adoption of DXR-CAD for TB screening and triage [13]. We sought to include at least two healthcare workers (HCWs) per site, including radiographers, radiography technicians, radiologists, and other clinicians involved in routine care provision. This was achieved for all sites, except Nigeria, where no HCWs were available. Interviews with HCWs included questions exploring their experience of use, regulations, and logistics (e.g., procurement, installation, etc.). To gain insight regarding programmatic considerations and implications, we also interviewed at least one program manager per site, given the limited number of eligible participants.
2.3 Data collection
From April to June 2021, ZZQ and RB who were both associated with the Stop TB Partnership TB REACH team, conducted individual and small-group semi-structured interviews with a total of 17 participants. In addition to these interviews, written answers were collected by an additional 11 participants.
Each interview was conducted using Microsoft Teams, due to pandemic-related travel constraints. Lasting 60–90 minutes, the questions tackled the programmatic context, technical characteristics, user experience, manufacturer service, and training, relevant to both CAD and DXR. A list of the interview questions is available in the topic list in the (S1-Topic List in S1 File). Extensive notes were taken during interviews by one interviewer and interviews transcribed verbatim from audio recordings. The interview topic list was revised several times to reflect the feedback and observations uncovered by the interview process. Interviews were conducted in English, except for the clinical staff from the pilot project in Vietnam. A questionnaire developed based on the topic list was sent to these participants through the program managers who translated the questions into the local language. The answers were collected verbally or by quick messaging tools in their local language and translated back to English by the respective program managers.
Questions that respondents could not answer in sufficient detail were forwarded to clinicians or radiologists from project field teams, who provided answers either by separate interview or in written form, depending on language requirements. Respondents were encouraged to provide supplementary photographs or documents to aid understanding. While respondents did not review transcripts, all respondents were followed-up by email to answer additional questions and for clarification, where necessary. Respondents were also given the opportunity to review and comment on the write-up of the results.
2.4 Data analysis
All transcripts, interview notes, and follow up questionnaires, were coded, using both deductive codes from the topic list, and inductive codes emerging from the texts. Data analysis was conducted using Microsoft Excel by ZZQ, RB and MMC, and supervised by CMD and JC, following the five-stage framework approach outlined by Pope et al [14]. The two interviewers agreed on initial codes (S2 Table 1 in S1 File) following a review of the transcripts and then independently coded three transcripts using this initial coding framework. Following discussion, codes were refined and amalgamated to form the final framework. The coded material was organized into themes using thematic analysis [15–17]. Table 2 shows an example of the coding process.
Six themes were identified, with 22 secondary and tertiary codes. The team discussed the themes, and existing theories or frameworks that could guide presentation of the data. From this, the Consolidated Framework for Implementation Research (CFIR), a conceptual framework which aims to promote the systematic analysis of implementation studies [18], was used to organize and present findings as barriers and facilitators along the CFIR constructs of adaptability, complexity, inner setting (e.g., readiness for implementation, knowledge, and beliefs of users), policy and process. Definitions of each construct concept used in this study and how they were mapped onto the final codes are presented in S3 Table 2 in S1 File.
3 Results
We present the perspectives of 26 respondents from six pilot projects spanning different use cases for DXR-CAD for TB screening and triage. Participants had the following roles: seven project coordinators or managers, three technical or research staff, three radiographers, five radiologists, and eight other clinicians. As these technologies are relatively new, all projects had implemented DXR-CAD for less than a year at the time of interview (Table 1). The themes are presented in terms of facilitators and barriers within the different levels of the CFIR and summarized in Table 3.
3.1 Theme 1: Increased portability involves compromises on other device characteristics
Portability was the most frequently mentioned advantage of UP DXR machines, which participants often compared favourably to stationary and heavier-duty mobile X-ray systems.
“Regarding the X-ray machine in general, they were quite happy with the portability. One guy took me to the COVID-19 isolation ward and showed me the ‘portable’ X-ray they were using; this was actually a huge X-ray machine and it had to be dragged […] and was […] over 100kg.”–Project Director, Pakistan
Nevertheless, despite marketing claims that the product can be hand-held (the Xair generator) or carried in a backpack (Delft Light), participants indicated that this applied to only some components, while other aspects, such as the detector panel, console and critical accessories, including the supporting frames and lead apron, were not truly “hand-held” or “back-packed” during transportation.
“The equipment is not exactly a backpack; the weight of the complete set is nearly 70kg and cannot be operated by only one person in the field… Even these logistics come at an additional cost to us, for example, they might need a motorcycle to take it some places. However, even with this, it is more portable than the truck.”–Program Coordinator, Nigeria
The miniaturization of the hardware, and its resultant improved portability, came with compromises in meeting the needs of the project from the perspectives of respondents. These compromises included concerns with battery and throughput, internet dependence, manoeuvrability and stability, and image quality and suitability for patients with diverse body sizes; in our analysis, these are categorized as sub-themes and discussed below.
3.1.1 Battery and throughput.
Limited battery capacity and throughput were mentioned almost unanimously by respondents as the primary disadvantage of using the UP DXR system. The X-ray generator was often mentioned as the rate limiting factor, with the projects in Zambia and Vietnam, who both used the Fujifilm Xair machine, reporting less than 100 shots on one full battery charge. However, other projects received updated versions of the Xair device and reported increased battery capacity. Participants across sites also mentioned that even when the emitter battery was sufficient, another system component, such as the CAD laptop, limited the throughput.
Several users, including all sites using UP DXR for extended periods of time, reported requiring external power sources, such as solar panels and power banks, to increase the number of exposures. However, some participants noted that this adds an extra component and weight, further compromising portability. The acceptability of lower throughput varied depending on the planned intervention aims. One participant noted that while throughput was lower than other equipment, it provided the capacity to offer services in locations they could not reach before, such as primary care level and remote locations:
“At first, I was disappointed, because I was coming from using other equipment that could screen 200 patients in a day without worrying about batteries. When we had 20–25 to start with, I was very disappointed. But now I see that with the lower throughput it still has an important utility–as I said, we were able to do screening at the facility level because of it.”–Project Lead, Zambia
3.1.2 Internet dependence.
Online or cloud CAD solutions are dependent on strong and stable internet connections, which may not be possible, particularly in remote areas. A slow internet connection can result in a long turn-around-time between image capture and CAD output. As a result, many respondents reported preferring to use offline or hybrid set-ups. A hybrid deployment method allows CAD processing offline, such that results can be synchronized online later, when a fast internet connection is available. Respondents noted that additional hardware is needed in offline or hybrid set-ups, adding weight and complexity to the set-up. For example, at two sites, respondents reported that the laptop with CAD installed provided was heavy, hindering portability.
Using an offline box eliminates the need for a second computer and is more portable, but this option was only available for some UP DXR-CAD combinations, as outlined in Table 1. Furthermore, a router is sometimes needed in addition to the other products, adding yet another component. A fully integrated X-ray system with CAD software–with no need for additional hardware–was not commercially available at the time of pilot implementation or while conducting interviews.
3.1.3 Manoeuvrability & stability.
Different choices of X-ray generator and detector supporting stands are available and often the more ‘lightweight’ options are offered by default. One respondent radiographer complained of the manual nature of the lightweight generator stand which was tedious when screening people of different heights:
“I think the other [problematic] component is the tripod stand. Usually, you discover that you have to alter the stand itself, raise it for the taller person and drop it for the kids. So, there’s this manual aspect of it.”–Radiographer, Zambia
Another participant switched to heavy-duty tripods, due to the instability of light-weight tripods which makes them prone to falls and damage.
“We just switched to more heavy-duty tripods because we had two falls. One caused by wind and the second by an accidental bump.”–Research Director, Vietnam
3.1.4 Image quality & suitability of patients with diverse body sizes.
Image quality was described as comparable to stationary machines, which can potentially facilitate adoption of UP DXR for case detection. A recent publication from the site in Vietnam showed slightly inferior image quality compared to stationary X-ray, but no effect on CAD interpretation or screening yields [19]. One clinician reported their preference for UP DXR over stationary devices in terms of image quality:
“Before we started using [the UP DXR], we had another similar [stationary digital] one […] I think the one that we’re using [now] has better quality images than the one we had before”–Clinician, Zambia
Just like stationary X-ray, most of the UP DXR systems offered a range of exposure settings for patients of varying size and in different positions. Respondents from one site mentioned taking images of patients lying down who were extremely ill with adequate image quality.
However, this was not the case for all devices. One limitation was related to the Delft Ultra devices, as they cannot be used in patients with a larger body size (above 100kg). As is described by one of the project staff:
“It produces unclear images when taking the person having a big body or being overweight, although images can still be usable.”–Radiographer, Ghana
3.2 Theme 2: Complexity increases with integration of hardware and software and interoperable data ecosystem
According to the CFIR, the perceived complexity of implementation is bound up with the duration, scope, radicalness, and disruptiveness of the technologies [18]. Respondents described levels of complexity in integrating UP DXR and CAD devices, as well as different electronic medical information systems:
3.2.1 Assembly and integration X-ray machine and CAD products.
The overall assembly process was described as easy after a while, including those that used a heterogeneous product combination. However, one brand of X-ray and CAD, Delft Light with CAD4TB, was noted to be more complicated to set up as it is composed of more devices and wires. The respondents noted to have to develop a packing list to avoid forgetting any hardware before going out to an outreach. Overall, the time required for assembly and disassembly ranged from 5–35 minutes depending on the UP DXR equipment used.
Participants in this study reported using six different combinations of four types of UP DXR and three types of CAD product, as outlined in Table 1. Product combinations from the same manufacturer or manufacturers with a formal partnership (e.g., Delft CXR with CAD4TB and FDR Xair with Lunit INSIGHT respectively) came pre-installed and integrated. However, for participants who used hardware and software from different manufactures, the integration required additional steps whereby both manufacturers were engaged before procurement to ensure that integration was feasible. In the Zambia site, the project lead reported that when using the Fujifilm FDR Xair with CAD4TB another laptop was needed to provide the link between Xair and the CAD4TB cloud. This was because the Xair console laptop was unable to run third-party software (including Google chrome) and so could not access the web browser CAD platform.
Sometimes an additional device was preferred, although not mandatory. For example, when using the Delft Ultra product, although the UP DXR generator included a screen for viewing the CXR result, the poor screen resolution meant the radiographer preferred to use another high-resolution device (either laptop or tablet) to view images.
3.2.2 Interoperability of the resulting X-ray screening data with other national health information systems.
One of the key challenges was the automatic exchange of CAD and X-ray data between different legacy electronic medical record or health information systems (HIS). Most CAD software providers include an application programming interface (API) that can easily link the X-ray and CAD results to the specific data system of the end-users. However, there is considerable heterogeneity in the data collected, and the HIS landscape is so fragmented that many countries prefer using locally built IT systems. To overcome this challenge, local expertise is needed for mapping of necessary data and database customization (working alongside local IT teams and the CAD and DXR providers).
As a result, diverse solutions emerged. One project site developed a bespoke data collection toolkit and integrated it with the CAD software. Another site used the Excel download from the CAD systems and manually transferred the X-ray CAD data to the national HIS. The other projects only used the locally built screening app or paper-based register and then manually entered the X-ray CAD data.
3.3 Theme 3: Limited capacity to implement technologies especially CAD software
Considering “readiness for implementation”, there was a perceived lack of knowledge and capacity to implement CAD technology among participants, especially regarding setting up and interpreting the threshold score. This despite the manufacturer-led training. In contrast, UP DXR was perceived as easy, partially due to its familiarity as an X-ray device, as opposed to novel CAD technology.
3.3.1 Capacity building.
Manufacturer-led training and installation of X-ray and CAD was conducted both online (remotely) and in-person for roughly equal proportions of participant projects. Following training, the X-ray process rarely presented difficulty, whereas CAD was seen as challenging, unfamiliar and built on ‘opaque’ concepts and mathematics. Reluctance to bring CAD into clinical decision making was commonly noted. Some users instead relied on clinical evaluation first and then the CAD score:
“When I’m interpreting an image, I usually look at the score last. Usually, I keep in my mind what the history and the clinical findings are, then I look at the image itself and what I’m seeing, and if what I’m seeing is highly suggestive of whatever condition I’m thinking of, then I correlate it with the CAD score.”–Clinician, Zambia
But not everyone is sceptical; some radiographers and radiologists were more open to using the new tool than clinicians, describing it as ‘high-tech’ in a positive way:
“I find that using it, the patients make you feel like you’re performing some kind of magic: this is a very simple device, yet making a chest X-ray out of it, you can see the surprise in their faces. It’s just technology doing its work.”–Radiographer, Ghana
However, some interviewees expressed concerns about healthcare workers using CAD’s prediction alone as the final diagnosis without prescribing any confirmatory testing, as expressed below:
“My worry, as it’s rolled out, and as the WHO has now approved it, is that clinicians may interpret a CAD score as TB and I suspect that we already see that […] in some places”–Program Lead, Zambia
To improve awareness of CAD technology, more theoretical training should be planned, especially for healthcare professionals including not only radiographers but also clinicians, radiologists and decision makers. While there is ample practical training for the radiographers, there is still a lack of knowledge of the theory behind CAD and uses of this technology, which was reflected in some scepticism reported.
3.3.2 Conservative application.
Regarding the CFIR construct of beliefs about the intervention, due to lack of capacity and limited understanding of CAD, current application of the technology is tentative in most cases. Caution is reinforced by varying levels of trust in CAD software, due to lack of familiarity, and fears that AI will ‘take over’, particularly among radiologists and clinicians, causing job losses. Only one site described trusting CAD to replace trained radiologists, where it was deployed for rapid triage by the screening camp coordinator. Most implementers were more conservative and required interpretation by a radiologist or other clinicians after CAD:
“On CAD you have mixed opinions, certain people really endorse it and get amazed by it, whereas some say you cannot replace a human [with] CAD, you don’t know the knowledge base that has been built up over years of experience.”–Project Director, Pakistan
Some participants were very impressed with the accuracy of CAD compared to experienced human readers. This, potentially facilitating further scale up of this technology.
“Our work shows that there are some software that are really good in our setting and that they perform on par with readers who have over 35 years of experience. I think it’s really nice and I’m hoping we can integrate this as a decision support tool for less experienced radiologists”–Research Director, Vietnam
The occurrence of perceived false positives and negatives, when compared with radiologist review mostly, was the main reason for questioning the accuracy of CAD decision. This was especially true in those with a history of TB. It is often said that the advantage of CAD is its ability to reduce inter- and intra-reader variability and supplement reader capacity when radiologists are too few, or absent. Our respondents concurred with this but mentioned the need to use their judgement to prescribe treatment.
“When someone has had TB before, the score will be high, when we know that the TB has healed and just left a scar, so sometimes we need to use our personal judgement on whether to initiate a person on treatment.”–Radiographer, Zambia
There were some individual fears of CAD software replacing human radiologists. This may be related to “conservative” national policies which continue to require radiologists be on site where CAD is used:
“In general there is hesitancy about the CAD software in Vietnam, the way its communicated in some of the guidelines is that it’s going to replace the radiologists and I think this is at odds with radiologists.”–Research Director, Vietnam
3.3.3 Threshold selection.
Threshold scores are used to decide if an individual should receive a follow-on diagnostic test or not. Consequently, the score will have a direct impact on sensitivity and the cost associated with follow-on testing. Increasing the threshold score will reduce testing demands (to match available resources and consumables); conversely, reducing the score will mean a more sensitive screening process.
In our study, the issue of threshold score selection strategy tended to divide respondents. A more experienced implementer engaged with the manufacturer to adapt the score in response to low yield. Only at one site did a user describe personally analysing data to select the threshold score (again, one of the most experienced implementers). In contrast, newer implementers seemed more likely to retain the manufacturer-recommended threshold score for some time, until enough confidence, and data, had been gained:
“I guess once we finish this study maybe we can edit the threshold and maybe after we’ve done CXR of a good number then we can recommend something.”–Project Director, Pakistan
One site operated at a threshold score that was selected by the country’s national tuberculosis programme based on TB prevalence screens. Additionally, a clinician reported combining CAD output with their own clinical decision-making:
“We don’t strictly stick to 60 and above [threshold], we just use this as a quick judgement when doing population screening. Symptomatic patients with a score of less than 60 are still screened”–Project Coordinator, Nigeria
3.4 Theme 4: Lack of guidance on radiation protection regulations suitable for UP XR machines
Considering the CFIR construct of “external policy”, there is a lack of international standards on UP DXR. With advances in highly sensitive X-ray detector technology, as well as imaging and noise-reduction technology, certain types of UP DXR can capture high-quality images with low-dose radiation. However, the international guidelines on radiation safety standards for UP DXR have not been updated accordingly: they remain based on standard, heavy-duty X-ray systems, presenting a barrier to implementation in the field.
Due to this lack of international guidance, national guidelines are often followed. In our interviews, respondents describe the national regulations, which vary widely between countries. For example, while one pilot project was allowed to operate outdoor without lead barrier, one project was not allowed to use UP DXR outside at all.
Similarities across country regulations included the use of measurements, being a common requirement for operators to wear dosimeters to measure exposure to radiation. Interviewees indicate that for some of the projects they have conducted measurements of scatter radiation and demonstrated to the local regulatory agencies that the accumulative radiation is much lower than the 1 mSv /year limit set out by the International Atomic Energy Agency [20]. In some instances, investigations were carried out by radiation authorities to approve the devices for use in the field.
In addition to customs clearance, approval from national radiation safety agencies was usually required for the import of X-ray machines into countries; different national radiation authorities have different policies on the approval of importation and use of UP DXR. National radiation safety standards vary; while some national authorities still opted to enforce full radiation safety measures and restrict UP DXR deployment to indoor use, others permitted the use of UP DXR outdoors without a lead wall.
“[the National Radiation Authority] didn’t want to give license until the whole room has lead walls, obviously you can’t have this in the field […] They looked at the catalogues, discussed with technicians and everything then they took dosimeters at different distances, 1m, 2m etc when using the machine, and took the dosage at every single place. And at the end they confirmed that if the human body is standing 2m away then they are safe and they can perform without the lead wall.”–Project Director, Pakistan
3.5 Theme 5: There is a need to both protect and share patient medical data
This theme refers to the dual requirements of sharing patient data with other clinicians and at the same time protecting the data to ensure patient confidentiality. There are two sub-themes at play: results storage (including back-up), and data privacy. Both fit under the CFIR concept “process: executing”, which means carrying out or accomplishing the implementation according to plan [18].
3.5.1 Results storage and back-up.
When using CAD to screen patients, patient data (including the CAD score output and CXR DICOM file) need to be transferred and stored securely. This is essential not only in terms of back-up, but also because CAD laptop storage can swiftly fill up: two respondents reported that this either slowed or prevented the operation of the whole system.
At the projects we surveyed, patient data was either backed up manually (to hard drives, at two sites) or to organizational servers (at one site) or it was automatically uploaded to a commercial cloud for storage (two sites). In one project with low throughput, no such data management systems were required. When using hard back-ups or backing up to the organization server, the implementing organization has more control over the security and privacy of patient data, while uploading to the commercial cloud is an automatic process.
However, the cloud back-up functionality brings benefits such as remote consultation with non-field-based staff to offer a second opinion. The project lead of the project in Zambia described this as one of the main advantages of using CAD.
“I felt for us that was very key. When you’re not able to print the X-rays, the radiographers have to be able to log in to the system to see them. Even for support, sometimes I log in from the office to give my opinion on specific X-rays. Sometimes the government radiologists also do this, we just tell them, and they log in and give us an opinion on an X-ray. For us that’s very key.”–Project Lead, Zambia
3.5.2 Data privacy.
Five sites reported the process of de-identifying data to comply with local data protection or privacy laws. When implementers were using commercial clouds, participants noted that CAD developers were able to facilitate de-identification whereas when implementers were storing data on their own servers, they often already had a system for removing identifying information.
3.6 Theme 6: Support and maintenance should be responsive to user needs
Regarding the CFIR construct “readiness for implementation: available resources”, the respondents reported several hardware faults happened within the first year of usage. Two teams that used Delft Light Backpack and Fujifilm Xair respectively reported the X-ray generator faults six and 11 months after installation. The Delft Light Backpack team also noted a solar panel fault and the long turnaround time to receive the replacement due to lack of in-country stockpile of spare parts. These faults delayed screening activities and service delivery and extra funds were needed to rent generators or other hardware pieces to mitigate the interruption of care delivery.
Additionally, two teams using different UP DXR systems, reported that after a few exposures, the X-ray system heated up, which slowed down its performance. All projects reported positive experiences with the suppliers’ remote support system. However, one of the project staff in Nigeria raised the concern that although suppliers offer 24-hour availability, access to this can be limited for implementers who work in field settings without an internet connection.
4 Discussion
This qualitative study identified barriers and facilitators to the implementation of DXR-CAD for TB screening from the perspective of early implementers. The technology was overall perceived as a tool to decentralize TB screening and triage in programmatic settings ranging from facility-based triage to hard-to-reach areas. Early implementers described facilitators such as portability, streamlined integration of products, training and support, as well as the potential to integrate CAD to the health information systems and varied data storage solutions. In contrast, barriers to implementation span the compromises that come with increased portability, namely battery power, internet dependence, manoeuvrability, and image quality for larger patients, the complexity in integrating DXR with CAD systems, limited capacity in implementing CAD, lack of international radiation safety guidance, the need to both protect and share patient data, and the need for comprehensive service and maintenance support.
The context and goals of any intervention will inform the acceptability of the UP DXR. While the relatively limited capacity of the internal built-in batteries with the X-ray system may be a limiting factor, this varies depending on the context of the intended use-case and setting. For example, in primary care facility-based settings, where low daily throughput is expected and electricity supply is expected, these technologies may be ideal. Where very high throughput is important, implementers may consider an alternate power source, supplementary use of a second UP DXR, as was described by one participant, or standard heavy mobile X-ray units, if the setting allows for these additions. In contrast, in very remote areas and where only battery power could be used, population is often dispersed, and small number of participants is often expected. In this case, the ability to access these hard-to-reach populations is the most important factor. Ultimately, balancing the compromises associated to greater portability depends on the goals of the project, and the priorities of the implementer team or program.
One important aspect of integration of CAD is considering the wider health system, and interoperability between national health information systems. The success of the roll-out of new technologies is dependent on an integrated health system alongside the actual technology: new innovations can only be “one part of that solution, necessary but not sufficient” [21]. Thus, to leverage the full capacity of DXR-CAD, enabling this interoperability between data systems is key. The digital nature of these tools presents an opportunity for the harmonization of digital ecosystems for TB surveillance. Most CAD software providers include an application programming interface (API) that can easily link the X-ray and CAD results to the specific data system of the end-users.
While most countries have a surveillance system, the high heterogeneity in the data collected, and the considerably fragmented HIS landscape pose challenges to follow the entire screening cascade and ensure TB treatment initiation (such as DHIS2 or eTB Manager) [22]. Measures to overcome this can include, but are not limited to, setting up basic standardized TB health data formats and elements and developing necessary API to allow data flow between systems. Local expertise is also needed for data mapping and database customization. CAD developers should expect to work with countries to develop workable and scalable models for key information exchange and adaptation to different countries.
Advances in UP DXR technology were accepted, perhaps because they build on radiological knowledge already well-established. In contrast, respondents revealed a mixed picture of the general acceptability of CAD technology, particularly due to fears of replacement and a poor understanding of software’s underlying algorithms. These findings align with a recent survey showing that while healthcare workers in the UK generally think artificial intelligence could be useful in medicine, they lack a full understanding of its underlying principles and are worried about potential consequences of its widespread use [23]. Current CAD training practices generally fail to address important individual, cultural or organizational assumptions that underlie attitudes to the technology.
These barriers are similar to described in experiences with other novel technologies; for example, scepticism among healthcare workers was reported when rolling out Urine LAM tests for diagnosis despite promising evidence [24]. The heterogeneity of feedback about CAD in this study could relate in part to the CFIR concept “tension for change”–the degree to which stakeholders perceive the current situation as needing change [18]–in this case the availability of radiologist readers. It is worth noting that clinicians and radiologists may be critical of CAD’s detection of false positives and negative, without critically considering that human readers are prone to similar errors. A clinician’s belief in their own capabilities to use CAD to make referral decisions (self-efficacy) can also determine how open they are to behaviour change using CAD. Furthermore, considering fears that CAD may replace radiologists, more could be done to evaluate use cases where CAD software is used as a decision support tool alongside human readers, rather as a replacement.
Finding an optimal threshold score for a given setting is presented a barrier to implementation for several respondents. Manufacturers sometimes propose a default threshold; however, as outlined by the WHO, manufacturer-recommended threshold scores are not always the most appropriate for the setting and use case and can result in unexpected and undesired false negatives and false positives [25]. Furthermore, some manufacturers do not propose a threshold, leaving implementers to choose. In these cases, and if it is not feasible or practical to conduct proper operational research to select the most scientific threshold score associated with a designated sensitivity or specificity before implementation, then the implementation should be accompanied with monitoring yield, screen positive rate, positive confirmation test rate among other factors, to then adjust the threshold accordingly. A standard operating procedure is needed to articulate and define the steps to change the threshold score. For example, an iterative operating point calibration (ITSC) mode was signed that allow TB programs to start with a rough threshold score and refine it through iterative cycles with new data during implementation until programmatic targets are reached [26]. By providing the opportunity to refine the threshold score in parallel to an ongoing intervention, ITSC is suitable for sites where time and resources do not allow for suitably rigorous research at the outset of an intervention.
4.1 Strengths and limitations of the study
This study is the first to assess the early users’ experience of DXR-CAD technologies for TB. This is partly a limitation because these technologies are new, and the respondents can only describe their own relatively brief experience during a pilot phase. Perceptions and acceptability may change as implementers become more familiar with these tools. Similarly, due to rapid change, some of the issues highlighted by respondents have now been improved by manufactures. Respondents from one project (Ghana) are affiliated with the manufacturer of the device, which is very new on the market. However, this was not the case for the remaining respondents. Finally, our study focuses on healthcare providers, hence no patient nor community perspectives are included in thi
4.2 Outlook for future implementers
There are several decisions that policymakers, country programmes and manufactures can take to ensure the successful implementation of these technologies. Firstly, there is a need to update international radiation safety standards to keep abreast of technological advances behind new X-ray devices with much lower radiation and higher dosage efficiency. Current international radiation safety standards should not constrain this technology. Given that any significant changes to international policy takes time, it is encouraging to see that some national regulatory agencies are taking bold initiatives to introduce this technology, with stringent monitoring plans in place to ensure safety; this alongside more data on radiation exposure as the use of the technology increases may lead to changing attitudes towards the safety of these devices.
The implementation of this technology is evolving rapidly, but several areas are clearly still open to improvement, especially the integration of X-ray machines and CAD software. In the projects surveyed, no CAD software can be installed directly on the X-ray console laptops. As a result, additional devices and connecting leads are required, particularly when the hardware and software are provided by different suppliers and for offline deployment. Software developers should partner with X-ray equipment manufacturers to further streamline the integration to reduce the complexity of assembly and the consequent transportation burden for users. Manufacturers of certain ultra-portable DXR systems should reduce the number of wires and hardware pieces to reduce the overall complexity of set up or assembly to improve user-friendliness.
Countries and programs thinking of using CAD technology should plan and budget for backing up and synchronizing the data ahead of time. As pilot projects are scaled up to national rollout, data, back-up and synchronization will become a bigger issue alongside issues relating to data privacy and security. Data privacy and security are clearly essential to protect the human rights of people affected by TB and others who provide their personal health data for processing using CAD technologies. Although this is of fundamental importance, it need not present a problematic hurdle. Safeguarding data privacy and security can and should be achieved without interfering with the use and functioning of CAD technologies. Implementers can use a range of legal, technical, and operational measures to safeguard the collection, storage, and processing of data while still ensuring the full functionality and benefit of CAD technologies.
Further, it is recommended that users should purchase a comprehensive and long-term warranty and support package [27]. The technologies are new and there have been malfunctions reported by these pilot projects interrupting screening activities and sometimes incurring extra costs, as outlined in our results. As the technologies are so new, early implementers have a role to play in advocating for product and service improvements by manufacturers, based on their on-the-ground experiences. We hope that our manuscript presented here will also encourage manufacturers to tailor their products to meet the needs of end-users and that the hardware and software will become more robust as the manufacturing matures with time.
5 Conclusion
This qualitative study has found that the respondents were able rapidly to use UP DXR and CAD following initial training, while the increased portability provide a great opportunity to decentralize radiological assessment. Further streamlining of the integration of X-ray and CAD equipment from different manufacturers and with different data would greatly enhance the user experience and reduce complexity. Two of the main barriers to uptake are the lack of understanding of CAD software–particularly threshold score selection–and the lack of international and national guidance on radiation protection suitable for UP DXR. Programmatic planners should seek guidance from their national radiation authority as early as possible and incorporate training to ensure capacity. Safeguarding data privacy and security can and should be achieved without interfering with the use and functioning of CAD technologies. Finally, programmatic planners should include comprehensive warranty and support packages.
Acknowledgments
The authors would like to acknowledge the contribution of all the respondents: Drs Monde Muyoyeta, Charles Imbuwa, Brian Shuma, S N, Vo Nguyen Quang Luan, Andrew Codlin, Kinz-Ul-Eman, Maurits Verhagen, Nicholas Setor Korto, Austin Ihesie, Bethrand Odume, Eze Chukwu, Kim Hee Jin, Kim Ji Yun, Kim Su Hee.
References
- 1.
World Health Organization. Global Tuberculosis Report 2022. World Health Organization, 2022.
- 2.
Stop TB Partnership. The devastating effect of the COVID–19 pandemic on the TB response -A minimum of 5 years of progress lost and 6 million additional people ill with TB. 2020.
- 3.
Stop TB Partnership. The Global Plan to End TB 2018–2022. 2019.
- 4.
World Health Organization. Module 2: Screening WHO operational handbook on tuberculosis Systematic screening for tuberculosis disease 2021 [26 March 2021]. Available from: https://apps.who.int/iris/bitstream/handle/10665/340256/9789240022614-eng.pdf.
- 5. Matteelli A, Rendon A, Tiberi S, Al-Abri S, Voniatis C, Carvalho ACC, et al. Tuberculosis elimination: where are we now? European Respiratory Review. 2018;27(148):180035. pmid:29898905
- 6. Frascella B, Richards AS, Sossen B, Emery JC, Odone A, Law I, et al. Subclinical Tuberculosis Disease—A Review and Analysis of Prevalence Surveys to Inform Definitions, Burden, Associations, and Screening Methodology. Clinical Infectious Diseases. 2021;73(3):e830–e41. pmid:32936877
- 7. Pande T, Pai M, Khan FA, Denkinger CM. Use of chest radiography in the 22 highest tuberculosis burden countries. European Respiratory Journal. 2015;46(6):1816. pmid:26405288
- 8. Qin ZZ, Ahmed S, Sarker MS, Paul K, Adel ASS, Naheyan T, et al. Tuberculosis detection from chest x-rays for triaging in a high tuberculosis-burden setting: an evaluation of five artificial intelligence algorithms. The Lancet Digital Health. 2021;3(9):e543–e54. pmid:34446265
- 9. Qin ZZ, Naheyan T, Ruhwald M, Denkinger CM, Gelaw S, Nash M, et al. A new resource on artificial intelligence powered computer automated detection software products for tuberculosis programmes and implementers. Tuberculosis. 2021;127:102049–. pmid:33440315
- 10.
ai4hlth.org. AI Products for Tuberculosis Healthcare [Website]. [cited 2022 9 March 2022]. Available from: https://www.ai4hlth.org/.
- 11. Fehr J, Konigorski S, Olivier S, Gunda R, Surujdeen A, Gareta D, et al. Computer-aided interpretation of chest radiography reveals the spectrum of tuberculosis in rural South Africa. Emily B Wong. 12(10):20–. pmid:34215836
- 12. Codlin AJ, Dao TP, Vo LNQ, Forse RJ, Van Truong V, Dang HM, et al. Independent evaluation of 12 artificial intelligence solutions for the detection of tuberculosis. Scientific Reports. 2021;11(1). pmid:34903808
- 13. Malterud K, Siersma VD, Guassora AD. Sample Size in Qualitative Interview Studies: Guided by Information Power. Qualitative Health Research. 2015;26(13):1753–60. pmid:26613970
- 14. Pope C, Ziebland S, Mays N. Qualitative research in health care. Analysing qualitative data. BMJ. 2000;320(7227):114–6. pmid:10625273; PubMed Central PMCID: PMC1117368.
- 15.
Qualitative Interviewing (2nd ed.): The Art of Hearing Data. Thousand Oaks, California2005. Available from: https://methods.sagepub.com/book/qualitative-interviewing.
- 16. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology. 2006;3(2):77–101.
- 17. Eisenhardt KM, Graebner ME. Theory Building From Cases: Opportunities And Challenges. Academy of Management Journal. 2007;50(1):25–32.
- 18. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science. 2009;4(1):50. pmid:19664226
- 19. Vo LN, Codlin A, Ngo TD, Dao TP, Dong TT, Mo HT, et al. Early Evaluation of an Ultra-Portable X-ray System for Tuberculosis Active Case Finding. Tropical Medicine and Infectious Disease [Internet]. 2021; 6(3). pmid:34564547
- 20.
International Atomic Energy Agency. Radiation Protection of the Public and the Environment 2018 [cited 2022 27 September]. Available from: https://www-pub.iaea.org/MTCD/publications/PDF/PUB1781_web.pdf.
- 21. Engel N, Wachter K, Pai M, Gallarda J, Boehme C, Celentano I, et al. Addressing the challenges of diagnostics demand and supply: insights from an online global health discussion platform. BMJ Global Health. 2016;1(4):e000132. pmid:28588980
- 22.
Stop TB Partnership, The Global Fund. Digital TB Surveillance System Assessment Report 2022 [cited 2022 9 September]. Available from: https://tbassessment.stoptb.org/.
- 23. Castagno S, Khalifa M. Perceptions of Artificial Intelligence Among Healthcare Staff: A Qualitative Survey Study. Frontiers in Artificial Intelligence. 2020;3.
- 24.
World Health Organization. Urine LAM tests: A golden opportunity to end TB deaths among people with advanced HIV 2019 [cited 2022 14 September]. Available from: https://cdn.who.int/media/docs/default-source/hq-tuberculosis/ias-2019-meeting-report.pdf?sfvrsn=d887ac84_7.
- 25.
World Health Organization. Determining the local calibration of computer-assisted detection (CAD) thresholds and other parameters 2021 [cited 2022 12 September]. Available from: https://apps.who.int/iris/bitstream/handle/10665/345925/9789240028616-eng.pdf.
- 26.
Stop TB Partnership. Screening and Triage for TB using Computer-Aided Detection (CAD) Technology and Ultra-Portable X-Ray Systems: A Practical Guide 2021 [cited 2022 27 September]. Available from: https://www.stoptb.org/resources-implementing-cad-and-xray/cad-and-x-ray-practical-implementation-guide.
- 27.
FIND. Digital Chest Radiography and Computer-Aided Detection (CAD) solutions for Tuberculosis Diagnostics: technology landscape analysis 2021 [cited 2022 9 September]. Available from: https://www.finddx.org/wp-content/uploads/2021/04/FIND-CXR-CAD-solutions-for-TB-diagnosis-7Apr2021-2pg-spread.pdf.