Content of the TPP

The completed TPP contains 28 characteristics organized into seven sections (S1 Document).

Participant demographics

Of the 110 people invited, the first round and second round received responses from 44 (40%) and 28 (25%), respectively, with 21 (19%) completing both rounds. The 51 total participants (46%) represented a variety of organizations and stakeholders with interests in global health diagnostics and surveillance, and experience in all WHO regions (Tables A and B in S2 Document). In each of the four fields of expertise surveyed, at least 30% of the group reported five or more years of experience, with HIS having the highest percentage of such respondents (53%) (Table C in S2 Document).

Delphi survey

In Round 1, the draft TPP (S3 Document) was met with basic consensus (> 50% agreement) for all 26 characteristics, but six minimal and 11 optimal criteria did not meet the preferred consensus level (> 75%), as shown in Fig A in S2 Document. We revised all of the characteristics that did not achieve preferred consensus and six additional characteristics, in total revising 18 of 26 characteristics, and we added one, based on common themes in the feedback.

In Round 2, this revised TPP (S4 Document) achieved preferred consensus for both minimal and optimal of all characteristics, as shown in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Agreement levels and distributions for the minimal ("Min") and optimal ("Opt") criteria of each TPP characteristic in round 2.

https://doi.org/10.1371/journal.pone.0228311.g001

Given the high level of consensus, a TPP consensus meeting as described in the Methods was not required, but it was deemed necessary to make three changes to address issues raised in the second round. The first two changes are minor and were not reviewed further:

1. Language support. Participants noted the importance of additional languages being part of the minimal criterion and the relatively low burden for an app developer to provide them. We increased the minimal languages to six to represent more of the variety used in LMICs, including languages with diverse characteristics that need to be considered in the app’s design.
2. RDT workflow support. In response to survey participants’ suggestions on having the app support running several RDTs at a time in a batch, we created this new characteristic. The optimal level requires support for batch and individual workflows, while the minimal can be met by an app limited to reading one RDT at a time.

The third change was more complex:

1. c. Security and privacy. Among the participants who participated in both rounds, this was the only characteristic whose optimal agreement was lower in the second round, and its minimal agreement fell in the second round more than any other characteristic (Fig B in S2 Document). Participants pointed out that the EU’s General Data Protection Regulation (GDPR) [23], while containing elements that can be useful for this product, cannot be applied as a whole outside the EU and, in fact, has elements which would conflict with the laws and policies of some LMICs. We consulted with each of the 10 participants who had commented in either round on GDPR, leading to a revision that adapts the six principles of GDPR to this product without requiring compliance with GDPR in its entirety. Put to a vote, this new text was approved by a vote of eight for and none against (two did not respond).

Additional feedback relating to certain characteristics has been summarized in Table D in S1 Document with the intent of providing useful information to product developers.

Intended use and accuracy

To yield as much data as possible for surveillance, one approach is to set the app’s capabilities so that it will not need a regulatory authorization as an in vitro diagnostic device. The current regulatory frameworks would likely limit an app that definitively determined results to the specific RDTs and phones used in its regulatory studies, precluding use with the hundreds of other RDTs and mobile phones in use. Some people would accept that restriction provided that such an app can improve RDT interpretation accuracy, but whether apps are more accurate than the eye has not been established. Some survey participants cautioned that interpretation of faint positives is at least as difficult for an app as for a well-trained person, particularly under this app’s diverse lighting conditions. By eliminating simple errors like overlooking a missing control line, however, an app may perform as well or better than real-world users with limited training and other demands on their attention. Such an app has not previously undergone a regulatory review. Therefore, product developers should aim to give health programs the ability to disable the result suggestion feature to adapt the app to in-country regulations.

Burden versus benefits

Improved surveillance depends on sustained use of the app as a reporting tool. The TPP’s brief characteristic on ease of use, which belies the complexity of that topic given the lack of baseline metrics from disparate settings, is but one part of this broader consideration. The app must offer enough benefits to justify its use. Current practice in some settings does not require reporting of negative results; no matter how user-friendly, no app can be less work than that. The more text and photos the user must submit to help the data recipients, the greater the challenge of getting busy healthcare workers to open the app for every test. The app should therefore be configurable by each health program to collect only the fields that program needs. Beyond minimizing burden, helping users care for their patients by interpreting tests accurately is a logical way to encourage use. When a result does have to be reported, the app can save time compared to paper by automatically collecting contextual data. The app’s data will also enable health programs to monitor the frequency of use and provide incentives for consistent use.

The Bring Your Own Device (BYOD) model

The issue of supporting mobile devices owned by users, or the related issue of supporting devices already deployed by health programs for other purposes, needs consideration. Supporting almost any device should promote the app’s adoption, but the diversity of devices in these settings will be a challenge to ensuring good performance. A physical component like an optical calibration card, by enabling performance checks and corrections for each phone, might solve that problem but create another: that of supplying and maintaining the cards in good condition. Ideally, the app would check each phone’s performance without needing any such accessories. Survey participants also expressed concern that too few users would have sufficiently capable devices for BYOD to be viable.

Personal data management

The handling of personal data concerning health requires special care and will need to be factored into the app’s development. One alternative is not to include patient names, patient identifiers, or any other personal data in the app. This would still enable surveillance but leave the problem of reporting results for patient records elsewhere. A second option is to structure this product to interoperate with other apps that take care of any personal data. Such a product, in the form of an app or a software library, would take care of the RDT reading steps, while another app would collect patient data and report results to a server, as some apps already do.

Diverse data policies

While stakeholders agree that this app must respect local approaches to data ownership, security, and storage, it is not simple to summarize those needs. Many jurisdictions have not published policies on them. Whether published or not, practices diverge considerably. Some countries will require that all data stay within their borders, for instance, while others will prefer to rely on the cloud. The app must be configurable to support the practices of each setting.

Pricing

As a nascent product category, an RDT-reading app does not have the established market and economic models of a malaria RDT or a tuberculosis test. Some participants called for the app to be free to use in these settings, as are some digital health software products and services. While donor funding may partially support the development of this app, the ongoing tasks of maintaining the app and supporting its users will have to be covered. An inexpensive pricing model, as suggested by other participants, would provide for this and could be part of a healthy market that encourages other app developers to offer innovations. Future work is needed to develop an approach that delivers value and is sustainable.

The primary limitation of the described work is a particular aspect of the survey approach. By requiring comments whenever a participant did not agree, which helped us understand the reasons for not agreeing, the survey could have encouraged agreement as the quickest route to completion. Consistent with that idea, when we made comments obligatory in round 2 for responses of “neither agree nor disagree,” the fraction of “neither” responses did decrease overall (compare Fig A in S2 Document and Fig 1), and agreement levels increased slightly among characteristics that had not changed (Fig B in S2 Document). Participant comments, a valuable output of the surveys, should be encouraged. Given the many optional comments submitted regardless of Likert score, however, it is worth considering whether it is appropriate to require comments only in cases of non-agreement.

We hope this TPP will promote the development of effective, useful, sustainable apps to support healthcare workers in interpreting and recording RDT results and to support healthcare programs in improved surveillance, particularly with a view on AMR-related data paucity.