External Quality Assessment of Reading and Interpretation of Malaria Rapid Diagnostic Tests among 1849 End-Users in the Democratic Republic of the Congo through Short Message Service (SMS)

Background Although malaria rapid diagnostic tests (RDT) are simple to perform, they remain subject to errors, mainly related to the post-analytical phase. We organized the first large scale SMS based external quality assessment (EQA) on correct reading and interpretation of photographs of a three-band malaria RDT among laboratory health workers in the Democratic Republic of the Congo (DR Congo). Methods and Findings High resolution EQA photographs of 10 RDT results together with a questionnaire were distributed to health facilities in 9 out of 11 provinces in DR Congo. Each laboratory health worker answered the EQA by Short Message Service (SMS). Filled-in questionnaires from each health facility were sent back to Kinshasa. A total of 1849 laboratory health workers in 1014 health facilities participated. Most frequent errors in RDT reading were i) failure to recognize invalid (13.2–32.5% ) or negative test results (9.8–12.8%), (ii) overlooking faint test lines (4.1–31.2%) and (iii) incorrect identification of the malaria species (12.1–17.4%). No uniform strategy for diagnosis of malaria at the health facility was present. Stock outs of RDTs occurred frequently. Half of the health facilities had not received an RDT training. Only two thirds used the RDT recommended by the National Malaria Control Program. Performance of RDT reading was positively associated with training and the technical level of health facility. Facilities with RDT positivity rates >50% and located in Eastern DR Congo performed worse. Conclusions Our study confirmed that errors in reading and interpretation of malaria RDTs are widespread and highlighted the problem of stock outs of RDTs. Adequate training of end-users in the application of malaria RDTs associated with regular EQAs is recommended.


Introduction
Although malaria RDTs are relatively simple to perform, they remain subject to errors [1]. Many errors are related to the postanalytical phase, i.e. wrong reading of the test and control lines and incorrect interpretation of the results [1,2]. External quality assessment (EQA) provides insights into performance levels of endusers and allows the assessment of the type and frequency of errors. However, delivery of high quality EQA panels to numerous RDT end-users may prove logistically challenging, especially in resource poor regions such as in many sub-Saharan countries, where malaria is endemic. Photographed RDT results may provide an alternative. Photographs have been used to assess the interpretation proficiency of HIV RDT performers [3] and to assess whether trained end-users of malaria RDTs had maintained their performance of accurate RDT reading [4]. However, large scale photograph-based EQAs assessing performance of end-users in reading and interpretation of malaria RDTs have not yet been organised.
In the last decade, Africa has witnessed an expansion in mobile phone use. Short message service (SMS) offers a cheap, simple and attractive mean of communication, access to large numbers of mobile telephone users and rapid data flow, also in a medical context. Examples of SMS based health interventions in Africa include projects on HIV/AIDS prevention [5], improved adherence to anti-retroviral therapy [6,7], accelerated communication of medical results [8] and accelerated communication of the malaria incidence and of stock data of malaria drugs [9,10]. Substitution of paper based reporting by SMS is expected to reduce data entry mistakes and may improve malaria control [11].
We here describe the results of an EQA on reading and interpretation of a three-band malaria RDT among laboratory health workers in the Democratic Republic of the Congo (DR Congo), a malaria endemic country where malaria RDTs have been introduced since 2010. The EQA consisted of photographed results of the RDT recommended by the National Malaria Control Program (Programme National de Lutte contre le Paludisme, PNLP). Individual laboratory health workers were invited to reply by SMS and a questionnaire was completed by the laboratory supervisor of the targeted health facilities.

Ethics Statement
The EQA followed an interpretive proficiency test scheme, was set up within the framework of routine diagnostics and capacity building purposes and was coordinated by the Institut National de Recherche Biomédicale (INRB), which is the reference laboratory of the PNLP. As indicated in ISO/IEC 17043:2010(E) (Conformity assessment -General requirements for proficiency testing, edition 2010, International Organization for Standardization, ISO Committee on Conformity Assessment, Geneva, Switzerland), the identity of participants was known only to people involved in coordinating the EQA programme, and was maintained confidential. Participation of laboratory health workers to the EQA was on a voluntary basis and their privacy was strictly respected. Following ISO/IEC 17043:2010(E), the Declaration of Helsinki, the Council for International Organizations of Medical Sciences (CIOMS) guidelines for epidemiological research, the guidelines of the American Anthropological Association, as well as the Belgian law of May 7, 2004 concerning experiments on the human person, and taking into account that all results of EQAs in Belgium are publicly available via the internet, submission of this EQA to the Institutional Review Board of the Institute of Tropical Medicine was deemed unnecessary.

Design of the Study
The EQA took place from May till July 2012 in DR Congo. It consisted of (i) 1 set of photographs with multiple choice questions (MCQ) for each laboratory health worker; (ii) 1 questionnaire per health facility and; (iii) leaflets explaining the EQA in detail and giving instructions on how to answer to the MCQ by SMS. The study coordinator at INRB shipped in total 2305 photographs and 1176 questionnaires to 14 co-investigators in 9/11 provinces in DR Congo. Provincial co-investigators were selected based on convenience, accessibility and personal contacts with the study coordinator or the PNLP. Co-investigators belonged to the National Tuberculosis Control Program, the Provincial Division of Ministry of Health and the Integrated Health Project (PROSANI). They distributed the photographs to laboratory health workers using malaria RDTs, and the questionnaires to the laboratory responsible of each corresponding health facility and gave the laboratory health workers a basic explanation about the study. Participating health facilities were members of the network of the National Tuberculosis Control Program, of PROSANI or of INRB, were provided with malaria RDTs by the PNLP or other organisations and were chosen by the co-investigators based on convenience, accessibility and personal contacts. Laboratory health workers answered the MCQ individually, by sending an SMS to the coordinator. Filled-in questionnaires were collected from the laboratory responsible by the co-investigators and shipped back to the study coordinator.

EQA Photographs with Multiple Choice Questions
Ten SD malaria Ag Pf/Pan (HRP2/pLDH, Standard Diagnostics, Inc., Kyonggi-do, Korea) test results were photographed ( Figure 1). Photographs were printed in real size on photographic paper (maco silk normal full colour CMYK, 300dpi, Bulckens, Herenthout, Belgium) and next to each photograph, a MCQ was provided. For each MCQ, six options were listed, numbered from 1 to 6. Per MCQ, only one option was considered correct. The SMS reply contained the following information (i) the code ''Eeq'' (for ''Evaluation Externe de la Qualité''), (ii) a 10 digit answer code consisting of the combination of the options (number from 1 to 6) for each of the 10 MCQs, (iii) the participant's name and first name, (iv) the health facility of the participant and (v) province. Participants were transferred 1 U.S. $ of phone credit. After closure of the EQA, an SMS message with the correct options for all MCQs was sent to all EQA participants.

Questionnaires
The questionnaire was filled in by the laboratory responsible. It

Data Entry and Analysis
SMS replies were transferred by Bluetooth to an Excel database managed by the study coordinator. If more than one SMS per laboratory health worker was received, only the first SMS was considered, which means that there was no system that could allow the laboratory health worker to re-call or correct a SMS sent with errors that he/she got aware of later. Ineligible answer codes shorter or longer than 10 digits or containing numbers higher than 6 were removed. To express competence, answers were scored as 1, 0.5 and 0 when they were categorized as correct or as minor and major error respectively (Table 1), and the sum of all scores was made (maximal score of 10). The Mc Nemar chi square test was used to compare proportions of correct MCQ answers between different questions.
Results of laboratory questionnaires which could be matched to at least one SMS, were entered in an epi-info database (Epiinfo TM , CDC, Atlanta, USA). The excel and epi-info databases were fused to make the link between the EQA score and health facility characteristics. To avoid correlation within health facility, one representative laboratory health worker for each health facility was selected (first one alphabetically). For bivariate analysis of MCQ scores associated with health facility properties, comparisons of median scores between 2 groups were performed using the Mann-Whitney Rank Sum Test. For multiple analysis of MCQ scores associated with health facility properties, a backward median regression model with MCQ score as outcome and the health facility properties from bivariate analysis as independents was built. For the covariates with missing values a missing indicator was added.

EQA
The study coordinator received 2484 SMS replies from 1892 laboratory health workers. After removal of 592 duplicates, 1849 answers (97.7%, 1849/1892) from 1014 health facilities were eligible, corresponding to 80.2% (1849/2305) of shipped photographs. The EQA was closed after 66 days and median time of SMS receipt was 21 days (interquartile range IQR 12-35 days). Most SMS (30.6%) originated from the province of Kasai Occidental ( Figure 2).
For each photograph, between 53.7 and 90.2% of correct answers were received (Table 1). Some errors occurred frequently. Invalid RDT results (photographs C and H) were not recognized by respectively 13.2 and 32.5% of participants, with 7.4% of participants missing the invalid result in both MCQs. Negative RDT results (photographs E and J) were not recognized by respectively 9.8 and 12.8% of participants, 5.6% of participants failed twice to recognize a negative result. Single faint or weak test lines (Photographs B, D and F) were reported as negative by respectively 31.2, 4.1 and 13.1% of participants and all three photographs were reported negative by 1.4% of participants. Test lines of weak intensity next to the control line appeared to be more frequently overlooked (9.0% more negative readings of photograph F than of D). Comparison of photographs with identical patterns but different line intensities further evidenced overlooking of low intensity lines: there were more correct answers for photographs I compared to A (p,0.0001) and for photograph G compared to B (p,0.0001). Participants encountered difficulties in discriminating Plasmodium falciparum from non-falciparum, and recognizing mixed infections. Photographs G and I, were read as positive but had wrong or no species identification for 12.1 and 17.4% of participants respectively (5.2% had wrong or no species identification for both photographs).
The median competence score for the 1849 laboratory health workers was 8.5/10 (IQR 7-9.5, Figure 3). All results were correctly reported (score 10/10) by 18.5% of end-users and satisfying scores of 9.5 and 9 were obtained by another 24.1% of end-users. 29.9% did not make any ''major'' error. Although no one scored zero, poor scores lower than 6.5 were obtained by 22.2% of participants.

MCQ Scores in Function of the Results of the Questionnaire
In bivariate analyses, the region of the health facility, the health facility level, the RDT brand and the RDT positivity rate were significantly related to the MCQ score (Table 3). Training, use of RDTs, time of experience with RDTs, and number of RDTs run per month did not influence the MCQ score.
For health facilities of the same level, from the same region and with the same positivity rate, facilities with training had a median MCQ score 0.5 higher than those without training (p = 0.0002). For health facilities with same training, positivity rate and region, the median MCQ score for high level health facilities was 1 higher than for low level facilities (p,0.0001). For health facilities with the same training, level and region, those with a RDT positivity rate .50% had a median MCQ score 0.5 lower than for those with a positivity rate, = 50% (p,0.0001). Health facilities with same training, level and positivity rate located in Eastern Congo had a median MCQ score 1 lower than those not in Eastern Congo (p,0.0001).

Discussion
The results of an SMS and photograph based EQA in DR Congo showed that reading and interpretation of malaria RDTs was poor for one out of four laboratory health workers. The most frequent errors were failure to recognize invalid or negative test results, overlooking of faint-intensity test lines and incorrect identification of Plasmodium species. Stock outs of RDTs occurred frequently. Half of the health facilities had not received RDT training, while only two thirds used the RDT recommended by the PNLP. After multiple regression, performance of RDT reading was better in comparable health facilities that received training, that were of higher level, that reported RDT positivity rates #50% and that were not located in Eastern DR Congo.
Some strong points of the SMS based EQA should be underlined. The low proportion of invalid results illustrates the simplicity of the SMS system already reported in other studies [11]. The power of mobile technology for EQA in a low resource setting is underlined by the number and the speed of SMS replies received in comparison with ''paper'' questionnaires. These findings confirm the potential of SMS based reporting to reduce turnaround time and loss rate compared to paper records [8,12]. Moreover, the SMS-based EQA appeared less susceptible to geographical limitations compared to the paper questionnaire, confirming the benefit of an SMS based system when results have to be transmitted over greater distances [12].
The EQA identified overlooking of faint or weak test lines as the most common error in reading and interpretation of RDTs, which could -in real life -lead to false negative results. The same problem has been observed by other authors [1,4]. Test lines, even faint, should be interpreted as positive [13]. Other previously reported errors were the interpretation of invalid tests [2,4,14] and lacking or incorrect species identification in a three-band test [2]. About one in ten participants interpreted negative RDTs as positive, which is lower than in EQAs for malaria microscopy [15,16]. Health facilities reporting RDT positivity rates .50%, scored lower in the MCQ, which might be an indication for overreporting.
Stock outs of RDTs are common in DR Congo, and this problem has been highlighted elsewhere too [17]. Avoiding stock outs therefore represents an additional challenge for RDT roll-out. SMS-based stock follow-up already has been applied for malaria drugs [9,10], and may be extended for RDTs [18].
In DR Congo, RDTs have been introduced since 2010 [15] and since April 2012, PNLP recommends their use in all levels of the health system. At the hospital or health center level, there was no apparent uniform strategy for malaria diagnosis. They mainly applied microscopy for malaria diagnosis, but the majority also performed RDTs. In line with PNLP recommendations, nearly all participating health posts -the lowest level of health care -use RDTs for malaria diagnosis while only a minority always performed microscopy. Of note, health posts relied most on RDTs, but had received the least training and scored lowest in the EQA.
Although the SD malaria Ag Pf/Pan test is actually recommended by PNLP, only two third of health facilities use this test. The Paracheck Pf two band test was previously recommended and was still used. The use of SD Malaria antigen Pf/Pv three band test by nearly 15% of health facilities was unexpected and is inappropriate, since Plasmodium vivax only rarely occurs in Central-Africa [19,20]. Use of other than the photographed test was associated with lower performance scores. This was particularly clear for Sud-Kivu and Maniema in Eastern Congo, where the recommended test was used in 16.9% and 0% of participating health facilities respectively. In addition, in Maniema only 8.3% of health facilities reported to have received training.
Improved RDT performance might be achieved through a combination of training, clear and simple bench-site job aids, supervision visits and organisation of further EQAs [1,17,21,22]. Of note, almost 50% of facilities stated not to have received training on the use of malaria RDTs, which implies that in these facilities, RDTs were introduced without training. Training is considered to be an essential part of rapid test implementation and WHO recommends national programs to include training in their RDT implementation plans and budgets [1,23]. Especially in a vast and resource poor country such as DR Congo, adequate countrywide training of health facility care workers may prove difficult, but distribution of well elaborated job aids, giving special attention to frequently made errors, might partially compensate for this. Malaria RDT job-aids and training manuals which may be down-loaded and modified as necessary are available on the  WHO website [23]. Following the present EQA, PNLP and its partners organised training on the use of malaria RDTs in more than 100 health zones in DR Congo, and updated their job aids. Supervision visits, including observation of health workers and addressing in particular frequent mistakes remain recommended [1,4] to complement an SMS based EQA system, assessing reading and interpretation performance only.
A number of limitations of this EQA should be considered. This EQA assessed only the malaria RDT reading and interpretation performance. Although this represents only one step of malaria diagnosis using a RDT, it is considered as a major source of errors [1,2]. As discussed above, the proposed EQA system is therefore just one part of the measures to be put in place to improve malaria diagnosis. Although participants were requested to answer to the MCQ individually, collaboration cannot be excluded. Statistical analysis was therefore based on 1 participant per health facility. Collaboration to solve the MCQ might however have had an educational effect for those participants with less experience. Since we had no full control over the spread of the EQA photographs, we could not determine the exact response rate and estimations are based on the total number of photographs and questionnaires shipped by the study coordinator. Some abuse of the EQA cannot be excluded, since participants sending a SMS to the coordinator received phone credit. Payment of an incentive or use of a toll-free number is crucial for participation, since lack of telephone credit is common [9,10]. Abuse was controlled for through listing of participants per health facility, and through verification of phone numbers and re-contacting suspect numbers. The mobile phone of the co-investigator was used to send SMS answers of participants in regions without telephone coverage (135/1849 answers from 3/ 14 co-investigators).
Retrospectively, the MCQ option ''positive for Plasmodium but I do not know the species'' might have been mixed-up by some participants with the option ''Plasmodium non-falciparum'', since differentiation between P. ovale, P. malaria and P. vivax is not possible when only the pan-pLDH test line is positive. Since this option represents loss of information on P. falciparum, we considered it as a minor error. Conversely, presentation of the EQA answers as MCQ options might have improved overall performance and might have had an educational effect.
In conclusion, organization of further EQA's seems indicated. It has been shown that participation to EQAs has an educational effect and raises performance, which may be explained, at least partially, by the feed-back given to the participants afterwards [16,24]. Taking into account that spread of smartphones and internet still remain limited in resource poor settings, distribution of paper printed EQA photographs will stay necessary. However, for dissemination of EQA photographs, future EQAs may take advantage of restocking of health facilities with RDTs or antimalaria drugs, limiting shipping costs and assuring reception. Further, the accompanying questionnaire should no longer be based on paper but rather transferred to SMS feed-back, which will increase matching between MCQ and questionnaires and decrease administrative work. In addition, an automatic feedback system to acknowledge delivery of the SMS should be foreseen [9], to avoid duplicate SMSing and anticipate the risk of health worker's ''reporting fatigue'' [11]. Although network fluctuations will remain a challenge for SMS-based EQAs, the rapid expansion of mobile network coverage and decreasing costs of mobile phone services offer opportunities to adapt this SMS and photograph based EQA system to other settings and apply it for other targets, such as HIV. Furthermore, recording of basic professional data through EQA allows future access to the end-users for malaria related issues.