Conceived and designed the experiments: FI DD KCK IO TA HA SS SOA KK. Performed the experiments: FI DD IO TA HA. Analyzed the data: KCK SS SOA KK. Contributed reagents/materials/analysis tools: FI DD KCK IO TA HA SS SOA KK. Wrote the paper: FI DD. Obtained and submitted data: FI DD IO TA HA. Reviewed data, manuscript, and provided guidance: KCK SS SOA KK.
The authors have declared that no competing interests exist.
The recent drive towards accreditation of clinical laboratories in Africa by the World Health Organization – Regional Office for Africa (WHO-AFRO) and the U.S Government is a historic step to strengthen health systems, provide better results for patients and an improved quality of results for clinical trials. Enrollment in approved proficiency testing (PT) programs and maintenance of satisfactory performance is vital in the process of accreditation. Passing proficiency testing surveys has posed a great challenge to many laboratories across sub-Saharan Africa. Our study was aimed at identifying the causes of unsatisfactory PT results in clinical research laboratories conducting or planning to conduct malaria vaccine trials sponsored by the National Institutes of Health (NIH).
PT reports for 2009 and 2010 from the College of American Pathologists (CAP) for the laboratories were reviewed as part of the process. Errors accounting for unsatisfactory results were classified into clerical, methodological, technical, problem with PT materials, and random errors. A training program on good clinical laboratory practices (GCLP) was developed for each center to address areas for improvement.
The major cause of PT failure in the four centers was methodological. The application of GCLP improved the success rate in the PT surveys from 58% in 2009 to 88% in 2010. It also decreased the error rate on PT by 35%.
A previous report from the CAP- PT participating laboratories indicated that the major causes of error were clerical. These types of errors were predominantly made in laboratories in the US, with much more experience in quality control, and varied significantly from what we found. In our centers in sub-Saharan Africa, methodological errors, and not clerical errors, accounted for the vast majority of errors. A process was started for continuous improvement which has decreased methodological errors by 35%, but more improvement is needed.
There are, currently, a large number of clinical drugs and vaccines trials being conducted in developing countries
The recent drive towards accreditation of laboratories in developing countries by the World Health Organization Regional Office for Africa (WHO-AFRO)
In addition to the provision of equipment from the developed countries, a resolute effort should be made to provide “hands-on” training to maintain a quality system
In PT or External Quality Assessment (EQA), samples are sent out periodically to registered laboratories to be analyzed and/or identified. Results from each laboratory are compared with those of the other participating laboratories in the group
The laboratories of the Noguchi Memorial Institute for Medical Research (NMIMR), Navrongo Health Research Center (NHRC) and Kintampo Health Research Center (KHRC) (all in Ghana) and the Center National de Recherche et de Formation sur le Paludisme (CNRFP) in Burkina Faso have been collaborating with the DMID in epidemiological studies and/or vaccine trials. These laboratories have been registered with the CAP for hematology, clinical chemistry and blood parasite surveys as a step towards continual improvement and accreditation. However, the laboratories had difficulties in consistently obtaining satisfactory scores at the beginning. Our aim was to identify the causes of unsatisfactory scores in the CAP PT surveys and to put in place measures to address these causes to improve performance in subsequent surveys.
A QA/QC advisor was hired in September 2009 by the U.S Naval Medical Research Unit No. 3 to coordinate the activities of all four trial centers. All PT results obtained in 2009 by the trial centers for General Chemistry, Hematology and Blood Parasites were reviewed, retrospectively. A standardized form was distributed to assist the centers with the investigational process (
Checklist of items for possible review |
1 - Instrument printouts/sheet agrees with submitted information? |
2 - Correct unit of measurement and decimal point? |
3 - Correct user group/method listed on submitted information/report? |
4 - Previous PT results show similar problem/shifts/trends? |
5 - QC result for 1 month before and after PT event show evidence of problems/shifts/trends? |
6 - QC record show changes of reagents, lot numbers or controls around the time of the survey? |
7 - Reagent and controls within expiration date? |
8 - Any other failures in this set? |
9 - Any training needs identified during discussion? |
10 - PT materials were retested and found to be accepted? |
11 - Consultation with manufacture indicates matrix effect on the samples? |
12 - Tech. re-read SOP (test method + Q.C procedure + reagent Handling) to confirm test method validity? |
13 - Follow maintenance table? |
14 - Last linearity of device was accepted? |
15 - PT materials investigation (handling, storage, analysis sequence, re-constitution and matrix effect)? |
Error | Cause(s) |
Clerical | 1 - Erroneous transcription of results from an instrument print-out or manual log |
2 - Reporting an incorrect unit of Measurement | |
3 - Reporting of an incorrect method or instrument | |
4 - Misplacement of a decimal point | |
5 - The selection of an incorrect reporting code | |
Methodological | 1 - Inappropriate use of QC materials |
2 - Using QC limits that are too wide | |
3 - SOP lack guidance on frequency of calibration | |
4 - Instrument used without performing test method validation | |
5 - Reagent problems | |
6 - Poorly written SOPs | |
7 - Procedure not in accordance with current standard of practice | |
8 - Lot-to-lot variation | |
9 - Inadequate maintenance | |
Technical | 1 - Inappropriate sample handling |
2 - Failure to calibrate pipettes | |
3 - Inappropriate dilution | |
4 - Water quality issues | |
5 - Improper reconstitution, preparation or mixing of PT materials | |
6 - Microscopic misinterpretation | |
Stability of PT | 1 - Improper storage conditions and/or delay in receiving |
Random | 1 - Any error that does not fall into any of the above categories |
In order to improve the PT performance for the calendar year 2010, the following GCLP and quality systems measures were taken:
We consulted users of various chemistry equipments in our countries and evaluated their performance in PT schemes, ease of use and maintenance prior to the purchase of new equipment. Uninterruptible Power Supply (UPS) systems were installed for all equipment to ensure stable power during operation. Subsequently, installation qualification (IQ), operator qualification (OQ) and performance qualification (PQ) were performed. The equipment qualification processes ensure that equipment is installed properly, operates as intended by the manufacturer, and continues to be suitable for its intended use. The procedures carried out IQ, OQ, and PQ included inter-equipment comparison, linearity, reproducibility & repeatability (R&R) and stability of control materials. Standard operating procedures for using and maintaining the equipment were prepared using information in the user manuals. Users were given adequate training on operation and user maintenance. A full-time biomedical engineer was employed for servicing the equipment. Levy-Jennings charts were reviewed to determine the proper frequency of calibration intervals. To ensure reagents used for our analysis were properly stored, surprise visits were paid to the suppliers and the importance of storing reagents at the required temperatures was explained to them.
We designed a process to verify clerical entries prior to final approval of PT results. The results were transcribed from instrument print outs to the PT answer sheet by one of the laboratory staff. Then a second laboratory staff will record the results from the answer sheet to the CAP website. Subsequently, the unit head or designee reviewed the entered results against the instrument print-outs.
Laboratory staff were advised to (1) select appropriate reagents, instrument and method codes; and to modify them if change of reagent, instrument or method occurred before the next PT shipment, (2) select the correct unit of measurement for each result entered, (3) select the correct code if the laboratory is not able to perform testing due to instrument malfunction or reagents shortage, and (4) ask for extension in rare occasions when the laboratory is not able to meet the deadline.
A one-week refresher training on preparation of quality blood smears and identification of blood parasites was organized for all of our centers at the Malaria Diagnostic Center, KHRC in Ghana. The malaria diagnostic center of KHRC was established as a center of excellence in 2008 as collaboration between KHRC, the Walter Reed project (Kenya) and the Malaria Clinical Trials Alliance (MCTA). KHRC conducted this training for the other centers before the first survey for blood parasites was received in 2010. During the training, microscopists shared their SOPs and discussed many issues related to malaria microscopy, such as QA/QC in the collection and preparation of blood smears, parasite identification and quantification. Pre- and post- test performances for each microscopist were evaluated.
The need to follow specimen handling instructions that come with the PT kit, use of calibrated pipettes and high purity water in preparation of chemistry reagents, calibrators and control materials was discussed with laboratory staff. For quantitative tests, we monitored the standard deviation index (SDI) from the PT summary and Participants’ Summary in relation to our test results. We aimed at having the SDI as close as possible to zero and our QC results as close as possible to the mean. If both SDI and QC results indicate high or low results then calibration or adjusting the calibration factor can resolve the issue. A high QC/SDI result means a QC value above +1SD and all five PT samples yielding positive SDI. A negative QC/SDI means a QC value below -1SD and all five PT samples yielding a negative SDI. However, if there was a positive SDI with a low QC results, the biomedical engineer and supplier were consulted, and they performed careful investigation. With the blood parasites PT, we redistributed misidentified slides or photographs to the microsocopists after grading of the challenge to allow them to have a second look. The participants’ summary discussion was used as an in-service training aid.
Extensive reviews of the user manuals for our equipment were performed. In addition, SOPs for the maintenance and daily operation of the equipment were written and made available to all personnel.
Our laboratories communicated with the airport officials the importance of keeping our samples at acceptable temperature during the paperwork clearance process. Moreover, if a shipment did not arrive within five days of the scheduled shipping date, a follow up process would be initiated. Usually, the PT vendor provided our laboratories with a tracking number to locate the shipment during transit. Once the shipment arrived, it would be analyzed as soon as possible.
When all the aforementioned causes of errors were excluded, we would classify the cause of failure as random error, especially when repeat testing indicated an acceptable performance. Our laboratories would not perform any corrective actions when random error was identified. Adjusting the testing system due to random errors can lead to future failures.
Methodological errors accounted for the majority of unsatisfactory results for both years in our laboratories (
Year | Methodological | Technical | Clerical | PT Stability | Random |
2009 | 73% (131) | 4% (8) | 17% (31) | 0 | 6% (10) |
2010 | 72% (115) | 4% (6) | 6% (10) | 16% (25) | 2% (3) |
2009/2010 average | 73% (246) | 4% (14) | 12% (41) | 7% (25) | 4% (13) |
CAP (2007) | 12% | 24% | 51% | 7% | 6% |
PT survey success rate in 2009 was 58% (14 of 24 PT shipments) while in 2010 the rate increased to 88% (28 of 32 PT shipments) (
Year | PT success rate | Average PT score |
2009 | 58% (14 out of 24 surveys) | 77% |
2010 | 88% (28 out of 32 surveys) | 90% |
Our surprise visits to reagent suppliers revealed poor storage conditions. Of the four suppliers visited, only one had temperature records. The recorded temperatures for the refrigerators were higher than the recommended 2–8°C due to overloading of the refrigerators. Room temperatures were higher than the recommended 20–25°C, as there was no air-conditioning in the reagent storage rooms.
Investigating causes of unsatisfactory performance and the immediate application of interventions are crucial steps in preventing future occurrences. Methodological errors were the major cause of unsatisfactory results in our laboratories. A previous report from the CAP PT participating laboratories indicate the possible causes of errors were 51% clerical, 24% technical, 12% methodological, 7% problem with PT materials, and 6% random errors
Specific items that were noted to contribute to failures include equipment selection, test method validation, equipment maintenance, reagent quality and storage, quality control procedures, continuing education, availability of expertise, and PT stability. Each of these items is discussed below:
Equipment related problems contributed greatly to the methodological errors. Proper selection and acquisition of the right equipment is the first step for a successful journey to quality
Once equipment is purchased, laboratories should ensure test method validation (TMV) is performed. TMV includes accuracy, precision, analytical sensitivity, analytical specificity, reportable range, and reference intervals
Once equipment is installed, laboratories should ensure proper scheduling, performance and documentation of daily, weekly, monthly, and other recommended maintenance
Poor storage and transport of reagents, controls and calibration materials by suppliers often leads to poor reagent performance that negatively impact the quality of testing. To eliminate these effects, our trial centers now procure reagents from reputable suppliers who showed proper storage condition and documentation. It is important that laboratories work with suppliers in improving service performance
QC must be tested and acceptable results obtained prior to release of patients results
Some of the errors encountered in the blood parasites surveys came from non-malaria organisms such as
The presence of a QA/QC advisor provided the centers with the hands-on technical support needed during clinical trials. Most of the supervisors in resource limited settings lack the technical expertise necessary to ensure accuracy of test results
We did not experience unsatisfactory performance as a result of unsuitable PT samples in 2009. This was, however, a significant contributing factor to the errors encountered in 2010. The PT stability error rate was about two times the value of the CAP data. Issues with PT survey materials stability are common in international laboratories. Many times the PT materials travel through several airports and warehouses before they arrive at their final destination. The temperature during the travel is not monitored and/or controlled. Some of the PT materials will require customs clearance before or after its arrival. Ensuring proper storage conditions during the paperwork processing is crucial to the stability of the PT materials. Some postal carriers, who lack refrigeration facilities at the airport, can accept refrigerators donated by laboratories to be used for storage of their shipment. However, this step will require a complete understanding from the postal carriers of the importance of compliance with temperature requirements. Laboratories should monitor the shipping calendar closely. Any delays of samples testing can negatively impact the quality of results. International laboratories do not have to order the PT product with less than ten days of stability
The intervention impact has been positive. All our centers had a higher average performance score in 2010 compared to that of 2009. KHRC did not have scores for 2009 because participation in the PT surveys started in 2010.
Improving the performance of laboratories in PT is a continuous process. Through a continuous program of education and training described above, we were able to achieve a 35% reduction in errors over the course of a year. However, despite a 35% reduction in methodological errors, this type of error remains the most common cause of errors at the four clinical trial centers in sub-Saharan Africa, accounting for 73% of errors. Comparing this rate to the US rate of methodological errors of 12%, emphasizes the need for more training of laboratory staff in this region in quality control methods that are well established in developed countries. Our centers will continue to use PT to improve Quality Assurance practices in our laboratories. We look forward to a continuous improvement process and sharing our experiences with other laboratories in the region.
The views expressed in this article are those of the author and do not necessarily reflect the official policy or position of the Department of the Navy, Department of Defense, nor the U.S. Government.
I am a military service member (or employee of the U.S. Government). This work was prepared as part of my official duties. Title 17 U.S.C. §105 provides that ‘Copyright protection under this title is not available for any work of the United States Government.’ Title 17 U.S.C. §101 defines a U.S. Government work as a work prepared by a military service member or employee of the U.S. Government as part of that person’s official duties.