https://orcid.org/0000-0002-3674-631X
Correction
27 May 2025: Pisani E, Rahmawati A, Mulatsari E, Rahmi M, Nathanial W, et al. (2025) Correction: A randomised survey of the quality of antibiotics and other essential medicines in Indonesia, with volume-adjusted estimates of the prevalence of substandard medicines. PLOS Global Public Health 5(5): e0004740. https://doi.org/10.1371/journal.pgph.0004740 View correction
Figures
Abstract
The World Health Organization warns that substandard and falsified medicines threaten public health in low- and middle-income countries. However, medicine quality surveys are often small and unrepresentative of the market, and the true scale of the problem remains unknown. We conducted a large field survey of essential medicines in Indonesia, and investigated how weighting survey results by market volume altered estimates of medicine quality. We collected 1274 samples of allopurinol, amlodipine, cefixime, amoxicillin and dexamethasone from the internet and a randomised sample of all outlet-types where medicines are sold or dispensed in seven districts across the world’s fourth most populous nation. We conducted compendial testing for identity, assay, dissolution and uniformity. Samples that failed any chemical test were considered substandard. We compared raw prevalence of substandard medicines with prevalence adjusted by the national sales volume of each brand, relative to its weight among survey samples. The weighted prevalence of substandard products was 4.4%, 47% lower than the raw estimate (8.2%). Only 0.5% of samples (unweighted 1.2%) deviated from permitted limits by more than 10%. More antibiotics failed testing than other medicines (weighted prevalence 8.5 vs 3.1; raw prevalence 13.6 vs 4.9, both p<0.000). We found no relationship between quality and price; branded status; public procurement status; or outlet type. In Indonesia, unweighted survey data appeared to substantially over-estimate the health threat posed by substandard or falsified medicines. The types of sampling bias that led to over-representation of poor quality products in our survey are also indicated in other published surveys, possibly exaggerating the scale of the threat to public health posed by substandard and falsified medicines globally. Weighting survey results by sales volume likely improves robustness of estimates of medicine quality measured in field surveys.
Citation: Pisani E, Rahmawati A, Mulatsari E, Rahmi M, Nathanial W, Anggriani Y, et al. (2024) A randomised survey of the quality of antibiotics and other essential medicines in Indonesia, with volume-adjusted estimates of the prevalence of substandard medicines. PLOS Glob Public Health 4(12): e0003999. https://doi.org/10.1371/journal.pgph.0003999
Editor: Elize Massard da Fonseca, Getulio Vargas Foundation, BRAZIL
Received: June 26, 2024; Accepted: November 11, 2024; Published: December 12, 2024
Copyright: © 2024 Pisani et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Additional data are available in three locations, all within the STARmeds repository, and identified by DOIs in the text or references. Code and supplementary data for this specific paper (including the weighting and analysis code in Stata format, the supplementary methods description, supplementary tables and author contributions) are at: https://doi.org/10.7910/DVN/QRKDWG Data and documentation related to STARmeds fieldwork more generally are in the study archive. This archive is easiest to use in Tree view. It contains the sample level data produced by the STARmeds field study, including all of the raw laboratory data, in csv format. Also included are laboratory protocols and a more detailed description of methods. The archive can be accessed at: https://doi.org/10.7910/DVN/RKYICP. Finally, we provide a free toolkit to help researchers and regulators design and implement medicine quality field surveys using mystery shoppers. The toolkit contains downloadable and adaptable versions of data collection software, field control forms, field worker contracts and other potentially useful documentation. The Toolkit can be downloaded from: https://doi.org/10.7910/DVN/OBIDHJ.
Funding: The study was funded by UK taxpayers through the UK Department of Health and Social Care and the National Institute for Health Research, under NIHR Global Health Policy and Systems Research Commissioned Awards (https://www.nihr.ac.uk/), grant number NIHR131145. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: Yusi Anggriani is a member of the Indonesian Ministry of Health’s advisory committee on medicine pricing, and a member of the World Health Organization Technical Advisory Group on Pricing Policies for Medicines. Elizabeth Pisani has worked as a consultant on research commissioned by the WHO Incidents and Substandard/Falsified medical products team. All other authors report no conflict of interest. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Introduction
In 2017, the World Health Organization (WHO) estimated that 10.5% of medicines in all low- and middle-income countries were substandard (they did not meet the standards laid out in their market authorisation paperwork) or falsified (they deliberately misrepresented content, identity or source) [1]. If correct, this figure should be alarming to those striving to expand access to medicines for 7.3 billion citizens of these countries.
The 2017 estimate was based on a review of 100 studies, mostly of malaria or TB medicines sampled from retail outlets in sub-Saharan Africa (see study archive for extracted data [2]). More recent reviews have found similar results, with overall aggregate prevalences of poor quality antibiotic, cardiovascular and diabetes medicines in low- and middle-income countries of 17.4, 15.4, and 10.8% respectively [3–6]. All note that prevalence reported in published studies are unlikely to be generalisable because of small sample sizes, unrepresentative study designs and variations in medicines included; tests performed; and pharmacopeia and definitions used. No reviewed study considered market share of different brands when calculating results.
Global health actors continue to call for survey data that would provide a more nuanced understanding of the actual prevalence and distribution of poor quality medicines circulating in specific markets [7, 8]. But the cost of testing medicines, combined with methodological challenges, mean truly representative sampling is rarely feasible [9, 10].
The STARmeds study, reported here, developed sampling and weighted estimation methods designed to increase the representativeness of medicine quality survey data in resource-limited settings.
Materials and methods
All methods are described in greater detail in S1 Methods, according to MEDQUARG guidelines [10]. That document provides details of secondary data sources, medicine and site selection, sample size calculation, sample frame construction, sample collection and handling, data entry and management, laboratory testing, ethics protections, and estimation methods. We summarise these briefly here. Additional data, including a MEDQUARG reporting checklist, estimation code, laboratory protocols, full sample-level data and codebook, and a free, downloadable toolkit for conducting medicine quality surveys are provided in the study archive [11, 12].
Study setting
Middle-income Indonesia is home to one of the world’s largest and most generous public health insurance schemes; it also has a vibrant domestic pharmaceutical sector. National health insurance now covers 249 million people [13]. Since inception of the system in 2014, national consolidated tenders pushed down the price of most medicines while increasing volumes consumed [14]. Medicines are provided free to insured patients, but administrative procedures are burdensome and many patients buy medicines outside the public system. In a nationally-representative survey of 345,000 households conducted in 2023, 80% of people who reported any symptoms of illness in the preceding month said they had self-treated. Only 35% of those with symptoms visited a health facility. Medicines accounted for just under a fifth of all out-of-pocket spending on health. Most of this spending was on modern rather than traditional medicines (42% on over-the-counter medicines and 39% on medicines bought with a prescription) acquired directly from health personnel [15]. Public procurement accounted for just 14.4% of the estimated US$3.6 billion spent on medicines in Indonesia in 2022 [16, 17].
Indonesian manufacturers warned falling prices may threaten medicine quality [18–20]. Cases of falsified vaccines in 2017 and contaminated paediatric syrups in 2022 added to public mistrust of medicine quality in Indonesia [21, 22]. However, in an independent field survey of cardiovascular/diabetes medicines collected from regulated and unregulated outlets in one district in 2021, all 204 samples passed testing for assay (which measures the percent of labelled active ingredient identified in the sample) and dissolution (the percent of active ingredient dissolved within a specified timeframe) [23].
Indonesia’s medicine regulator (Badan Pengawas Obat dan Makanan, BPOM) conducts extensive proactive post-market surveillance, though only from regulated outlets. BPOM tested 10,980 medicines collected through randomised sampling in the regulated supply chain in 2021. 3.7% were out-of-specification (tidak memenuhi syarat), meaning they failed at least one pharmacopeial test, were expired or damaged at the time of sampling, or were incorrectly registered or labelled. Another 2,559 samples were collected using risk-based sampling, focused on products of public health importance with high potential for quality defects; 5.2% failed on at least one dimension, for an overall estimate of 4.0% out-of-specification products nationally [24, 25].
Study design
STARmeds medicines were chosen based on public health importance, diversity of suppliers, risk of falsification and feasibility, in consultation with BPOM, the Ministry of Health, the national health insurer, the association of hospitals, and other stakeholders. In determining public health importance we considered the prevalence of the conditions treated by the medicine; the volumes procured in the public system and sold in pharmacies nationally; and the financial burden on the public health insurance system. See S1 Methods for details.
Included products are shown in Table 1; all require prescriptions in Indonesia.
Sampling locations were chosen purposively to reflect Indonesia’s geographic and economic diversity. We included all outlet types from which Indonesian patients commonly acquire medicines. Regulated outlets included pharmacies; public and private hospitals; primary health centres, and licensed phone apps. Unregulated outlets (technically forbidden from dispensing prescription medicines) included over-the-counter medicine shops, individual shops in bulk medicine markets; doctors; midwives and internet platforms. In each area, we listed and verified all outlets by type. In consultation with experts from Statistics Indonesia, we drew up a sample frame which took into account the relative size of each sampling location, and specified the desired number of sampling venues for each district and outlet type. We then selected target outlets randomly from the list for each location and outlet type, using Stata 17’s random generation function.
All pharmacies and medicine shops were privately owned; numbers depended on the location. We selected a district public hospital (in most districts there is only one), one private hospital, and one public primary health centre (puskesmas) in each sampling area. All doctors and midwives included were in private practice. We randomly chose a total of five in each location. Further details are provided in S1 Methods. Table 2 shows locations, characteristics, sampling dates and type of outlet randomisation.
Sample collection and handling
Trained mystery shoppers bought medicines from selected retail outlets, using individualised sample frames which specified medicines, doses and a price point (cheaper or more expensive) but not a brand. Shoppers carried prescriptions, offering them if requested. In health facilities, study staff sampled overtly, buying a branded and an unbranded version of each study medicine if available.
Each sample was stored in a separate, pre-barcoded bag. Field data (barcode, geolocation, price) were entered on smartphones, using KoboCollect software [26]. Samples were delivered daily to local study hubs. They were inspected for anomalies in packaging or labelling. Data entry staff entered additional product details and took high-resolution photographs using tablets pre-loaded with the study software. Samples were stored with a temperature logger in an airconditioned room until transfer to the laboratory.
Laboratory testing
Samples were tested using United States Pharmacopeia (USP) reference standards, at PT Equilab International in Jakarta. For cefixime capsules, we followed Supplement 1 of Farmakope Indonesia 6th [27]. All other medicines were tested according to USP 43, NF38 monographs [28]. The acceptance criteria for each product are shown in Table 3.
Identification, assay and uniformity of assay were tested using high performance liquid chromatography (HPLC-UV Waters, Alliance 2695 with UV Detector 2489). HPLC was also used in dissolution testing for amoxicillin tablets and dexamethasone. Dissolution of amoxicillin capsules, allopurinol, amlodipine and cefixime was analysed using Spectrophotometer-UV/VIS (Shimadzu UV-1800). No dissolution testing was performed on amoxicillin dry syrup formulation. Full laboratory protocols are in the study archive.
We could not afford to test for impurities.
Product verification and falsification
We provided all 79 market authorisation holders with per-sample data and high-definition photos of primary and (if available) secondary packaging, asking them to verify all sampled batch numbers, expiry dates and maximum retail prices against their production records. Samples with confirmed anomalies were considered falsified, as were any with no or incorrect active ingredient.
Product quality
We defined a substandard sample as one that failed any pharmacopeial test, using the limits shown in Table 3.
Raw prevalence was calculated by dividing the number of substandard samples by the number tested.
Previously-suggested measures of "extreme deviation" do not take into account the variations in assay values permitted by most pharmacopeia [30, 31]. We calculated a single measure of deviation from permitted values as the maximum among the following:
percentage points by which assay exceeds or falls short of the limits of acceptability;
percentage points by which final dissolution value falls below the permitted threshold.
To distinguish between marginal and more extreme deviation, we also report prevalence of samples that deviate from permitted levels by over 10%.
Weighted estimates
Outlets were chosen randomly, but mystery shoppers could not credibly request a list of individual medicines by volume for truly representative sampling. In our adjusted estimates, we thus weighted each sample in our study by the ratio of a brand, dose and formulation’s market share to its weight among our study samples, using the total market for all five medicines as the universe, as follows:
For estimates of prevalence by medicine, we recalculated the relative volumes within the universe of each active ingredient. Details are provided in S1 Methods.
Volume by product (active ingredient, formulation, brand and dose)
For calendar 2022, pharmaceutical data aggregator IQVIA listed non-zero sales volumes in the Indonesian market for 467 separate versions of the medicines and formulations included in the study. Each was a unique combination of medicine, formulation, dose and brand (or, for unbranded generics, market authorisation holder). We used these volumes as the basis for our estimates. For public procurement products (n = 17), we added national public transaction volumes for calendar 2021 to IQVIA volumes, adjusting to avoid double counting in the hospital sector. We redistributed IQVIA’s unnamed "generic manufacturer" volumes across unbranded generics as described in S1 Methods, which also explains how we imputed volumes for 18 products collected in our survey but missing from IQVIA data.
Price-related analyses
When investigating the relationship between price and quality, we used the price paid for the specific sample, per smallest counting unit. Samples of public procurement medicines acquired at no cost were priced at the public procurement cost plus a government-permitted margin of 28% for tax and handling.
Ethics and reporting
The study design was widely discussed with BPOM and a multisectoral national working group on medicine quality. The study protocol was approved by institutional review boards at Universitas Indonesia (970/UN2.F1/ETIK/PPM.00.02/2020) and Imperial College London (21IC7265). We also obtained written permissions from the health department or other competent authority in each sampling district.
Dedicated study staff in the district research hubs provided full-time problem-solving support by phone to mystery shoppers. We immediately notified BPOM of any suspect product; we shared sample-level results for all study products once certificates of analysis were issued.
Results
We collected and tested 1,274 samples, as shown in Table 4; 82 were not of targeted doses. Four locally-made branded products were registered by multinational companies. The rest were registered by 75 different Indonesian pharmaceutical firms; medicines were made by 72 different manufacturers, all in Indonesia. (This excludes two internet-acquired samples of locally registered brands apparently packed for and imported from other markets.) We collected one originator product, 178 unique medicines sold under proprietary brand names (branded generics), and 101 unique unbranded products, differentiated by market authorisation holder. Table 4 shows the distribution of tested samples by sampling area and medicine, and the number of unique products. Table 5, meanwhile, shows the distribution of samples by location and type of outlet.
Branded medicines dominated the online samples (72%); in physical outlets, 51% of samples were branded.
Raw quality estimates
All samples contained the correct active ingredient. Table 6 shows further test results, and the number confirmed falsified among the 578 samples for which we received confirmation from market authorisation holders.
Overall, 8.2% of samples failed at least one pharmacopeial test, with significant differences by medicine type. Samples expired at most recent testing date (n = 47) were no more likely to fail testing than unexpired samples (8.5% vs 8.2%, p = 0.95). The anti-hypertensive medicine amlodipine, the only chronic disease medicine in the study, had the lowest testing failure rate, at 1.6% (all failing in uniformity of content only). The antibiotics amoxicillin and cefixime had the highest testing failure rates (10.0% and 19.0% respectively, totalling 13.6%, compared with 4.9% for non antibiotics, p<0.000).
Of 72 manufacturers, 30 (41.7%) made at least one of the substandard samples.
Most of the failures were clustered close to permissible quality limits, (depicted schematically in Fig 1 as the zone between the dotted lines). Fifteen samples (1.2%) from 11 manufacturers, deviated from permissible limits on assay or dissolution by 10% or more; only 1 of these was confirmed falsified by its market authorisation holder.
Falsified medicines
No sample was deemed falsified based on test results. By December 2023, 45/79 market authorisation holders had checked sample details against manufacturing records, covering 578 samples; 21, from 12 companies, were confirmed falsified (3.6%). Seven of the falsified samples failed pharmacopeial tests (33.3%), compared with 9.4% of samples with confirmed correct production records (p <0.000).
20/21 falsified products were branded; six had fake batch numbers, while 15 had incorrect expiry dates. The majority of falsified samples (15/21) were purchased from unregulated internet sellers; only one was acquired from an outlet type permitted to sell prescription medicines. Study staff flagged 11/21 confirmed falsified products as suspicious at visual inspection.
Source of medicines
In regression analysis, after controlling for differences in medicines, the odds of testing substandard were 2.2 times higher for samples collected in rural areas, compared with those collected from physical locations in cities (p<0.001). There was no significant difference in the pharmacopeial quality of samples acquired from different types of outlets. 8.0% of online medicines failed testing, against 8.3% acquired from physical sources (p = 0.83). However, among samples acquired online, those bought from individuals selling on general marketplaces or social media were more likely to fail any pharmacopeial test than those bought from licensed online vendors or verified online stores of physical pharmacies (11.3 vs 3.6%, p = 0.01).
Samples bought from the types of outlets included in BPOM’s routine post-market surveillance were no more likely to fail testing than excluded outlets (7.8% vs 8.9%, p = 0.46).
Branded status and price
Branded and unbranded products were equally likely to fail testing (9.1% vs 7.2%, p = 0.23) including after controlling for differences in medicine, district, or source. In unweighted analysis, medicines available free to patients in the public insurance system were significantly less likely to fail testing than medicines paid for out-of-pocket (3.8% vs 8.9%, p = 0.031).
Prices varied widely, both by brand and by outlet. In logistic regression, there was no relationship between price and pharmacopeial quality, including after controlling for differences in medicine, district, or source. The relationship between price and quality is reported in detail elsewhere [32].
Weighted quality estimates
We collected at least one sample of two thirds of the registered medicines and doses specifically targeted in our sampling frame (for example amlodipine 5mg), as well a third of products of other doses of the same formulations (e.g. amlodipine 10mg). In terms of market share, however, the samples we collected represented 97.1% of all targeted doses and 89.9% of other doses. Overall, we collected at least one sample of products representing 95.4% of the total Indonesian market for the five medicines and formulations included in STARmeds.
We weighted our survey results using the ratio of a brands’ market share to its share of the total number of samples in our survey. Fig 2 plots the number of samples of each product tested (by pass/fail status) against its weight; S1 Table lists the respective market shares and sample shares, and gives weights.
*Market share is calculated separately for the unregulated online market. Graph omits 4 samples with market share 16–76 times their share in the STARmeds sample; all passed testing.
Overall, 88% of the unique products in our sample (shown to the left of the dotted line in Fig 2) were over-represented, compared with their market share. This was partly because we deliberately sought unregulated online sellers who sell tiny volumes; the 14.6% of study samples from these sellers represented just 0.0022% of national market volume for study medicines. Additionally, we oversampled active ingredients with smaller distribution to allow for inter-medicine comparisons. For example, cefixime accounted for 15.3% of our sample, but just 3.7% of the market of our universe of 5 medicines. Failure rates were higher in products that were over-represented in our sample, compared with under-represented products (9.8% vs 3.3%, p<0.000).
As Fig 3 shows, adjusted prevalence of substandard samples across products tested in the study was 46.9% lower than raw prevalence (4.4% vs 8.2%). When we weighted samples by their relative market share within the universe of a single medicine (also shown in Fig 3), the prevalence of poor quality amlodipine fell by two thirds, while amoxicillin dropped by just 13%. While in unadjusted analysis public procurement medicines appeared less likely to fail testing compared with other medicines, the relationship was reversed, and lost statistical significance, once adjusted for the high volumes of public procurement medicines.
Red value labels denote statistically significant differences in analysis of raw data; green labels note differences significant in the adjusted analysis.
The weighted prevalence of medicines failing assay or dissolution by more than 10 percent of the permitted margin was 0.5%.
Discussion
We collected 1,274 samples of five medicines across four therapeutic groups from all sources from which Indonesian patients commonly acquire medicines, and tested them for identity, quantification and uniformity of active ingredient, and dissolution. The raw prevalence of substandard products was 8.2%. However, when we adjusted for market size of individual brands, the estimate fell by 46.9%, to 4.4%. This suggests that poor quality medicines do not, in general, achieve large market shares. Using the raw rather than the adjusted estimate, we risk overestimating the volume of substandard study medicines sold in a year in Indonesia by 75 million units.
Our findings are consistent with the one other study we found which adjusted quality estimates for market size [33]. In that study of an antimalarial, an antibiotic and an analgesic in the Democratic Republic of Congo, the raw estimate of poor quality medicines (27.2%) fell to 1.3% when adjusted for market size data provided by distributors. (The authors acknowledge that, since samples were collected from distributors, these measures do not reflect degradation and other loss of quality in the supply chain.) Many academic studies of medicine quality included in recent metanalyses prioritise brand variety in their sampling [1, 5, 6, 34]. We therefore believe other, unadjusted studies may similarly mis-represent (and possibly overstate) the true proportion of medicines that are of poor quality.
Our adjusted estimate for five medicines was 10% higher than the 4.0% found by the Indonesian regulator BPOM in nation-wide post-market surveillance of over 10 times as many samples across all medicines in the months before the study [35]. STARmeds sampling strategy and definitions of quality did not mirror BPOM’s exactly; the differences could affect the results in both directions. We included outlets not licensed to sell prescription medicines, and used mystery shoppers rather than uniformed regulators to sample in retail outlets. Both factors could increase our chance of collecting poor quality products, compared with BPOM’s strategy, although in fact, we found no significant difference in quality between samples collected from outlet types sampled by BPOM and others. (Such similarity between regulated and informal markets was also recent reported in Nigeria [30].) Other differences are more likely to decrease the relative chances of sampling poor quality products. We excluded sterile formulations at higher risk for production errors, and psychoactive medicines at greater risk for falsification, and did not include labelling errors in our definition of substandard products.
Despite these differences in definitions and sampling, the weighted estimate in our study is very similar to that published by the national regulator for the Indonesian market as a whole, yet it was achieved with a sample that was less than one tenth of the size. A full consideration of the cost of surveillance is published elsewhere [36], but we suggest that weighting raw estimates by market size is likely to be a resource-reducing option for arriving at a better understanding of the magnitude of the threat posed by substandard or falsified medicines in national markets. Sources of data on sales volumes vary by country. Data from commercial aggregators, where available, are expensive. Other potential sources include data from distributors, wholesalers, public procurers and public insurers, as well as tax and customs authorities.
Even the adjusted figure implies a large number of substandard products in the Indonesian market—close to 87 million tablets, capsules or 5ml doses across 5 medicines. However, most failed samples hovered close to Indonesia and the United States’ permitted limits. Using one of the other 60 pharmacopeia used by national regulators [37], some may have met required thresholds. Since no pharmacopeia provides an evidence base for its chosen thresholds, it is impossible to estimate the clinical impact of samples deemed to have failed by a small margin. For many patients, especially in populations with lower body weights than those in standard-setting countries, it may be limited. We do not question the need for clear and well-enforced quality standards, but believe that a consistent and clear rationale for chosen pharmacopeial thresholds would help authorities prioritise measures of greatest importance to public health.
If quality is persistently low, the cumulative effect on a patient of even marginal deviations may be more severe, particularly in the case of therapy for chronic infections. Slightly lower than permitted dissolution of antibiotics may also place the product in the mutation selection window, favouring the development and spread of antibiotic-resistant bacteria [5, 38–40]. In our study, poor dissolution of cefixime is especially worrying, since WHO assigns it to the "Watch" category of antibiotics prone to resistance [41].
Our findings were comparable with other studies in Indonesia, Cambodia, India and Togo in finding that quality did not vary by price [42–44]. Researchers in Indonesia and other countries (including the United States, China and South Africa) have found that both patients and health-care providers distrust the quality of free medicines and unbranded generics; some suggest the distrust is deliberately sown to favour the financial interests of doctors, hospitals and pharmaceutical companies [21, 45–49]. Since the introduction of Indonesia’s national health insurance scheme, Indonesian media have also carried reports questioning the quality of its free medicines [50, 51]. We found no basis for selective distrust of free medicines or unbranded generics in Indonesia.
We succeeded in verifying production records with 44/79 companies. Only one confirmed falsified product was found in a regulated outlet permitted to dispense prescription medicines, again mirroring a recent Nigerian study [30]. Unexpectedly, the majority of confirmed falsified samples passed quality testing; most had expiry dates which did not match genuine batch numbers. This suggests criminals are extending dates on packaging of genuine products which continue to maintain serviceable quality beyond their authorised expiry dates.
Simple visual inspection by researchers familiar with brand packaging identified more than half of the products that were confirmed falsified. A pilot programme to train health care workers in Indonesia and Tanzania to spot and report suspect medicines was deemed promising and could be expanded [52]. However the signs would not be evident to most consumers, who do not have the same points of reference.
An important limitation of our study was that we could not afford to test for impurities. Even relatively well resourced regulators, including BPOM, do not commonly test for impurities, or for by-products of excipients; hence, the Indonesian regulator did not pick up the presence of lethal non-active ingredients in paediatric syrups during their extensive routine post-market surveillance in 2021/22 [53]. This underlines the need for more affordable medicine testing technologies, and greater attention to impurities and non-active ingredients in considering the quality of medicines [54–56].
Conclusion
Although our study is one of the most comprehensive single-country academic surveys of medicine quality, we found that the "headline" figure of 8.2% substandard medicines it yielded likely overestimated both prevalence of poor quality products and their threat to public health. When adjusted for brand volumes, just 0.5% deviated from permitted limits by more than 10%. The overestimate was caused by sampling techniques which over-represented brands with small market share, and medicines bought from unregulated sources. Other published studies do not report relative market share. However, the diversity of brands and outlets they report within small sample sizes suggests that sampling techniques commonly oversample marginal brands which, in our study, were more likely to fail testing. Many also sample from unregulated outlets, where falsified medicines are most commonly found. In our study, we found that informal markets and unregulated internet sources represent only a small proportion of sales, again leading to over-representation of falsified products in the survey compared to the market. This may differ in other markets; we encourage systematic research to quantify the relative size of different market segments in other countries.
In common with authors of recent systematic reviews, we believe the aggregation of results across dissimilar studies is misleading [3, 5, 6, 34]. While quality-assurance will differ in countries with different regulatory capacity and medicine import profiles, our study suggests that such aggregations may compound the effect of sampling biases. Together with binary pass/fail reporting of medicines which fail testing only marginally, this has probably led to an overestimate of the potential health threat posed by substandard and falsified medicines globally.
We recommend that regulators and others conducting post-market surveillance or primary surveys of medicine quality consider weighting by market size to improve understanding of the volume and distribution of poor quality products, and to plan and resource effective responses. We note, however, that even estimates weighted by market volume are expensive and time-consuming to generate [36]. Health authorities must assess the value of this investment relative to a case-finding approach with narrower focus on identifying and removing substandard and falsified products at highest risk of harming public health [57], perhaps considering market-wide, volume weighted surveys of prevalence every four or five years, with more targeted, risk-based surveillance in interim years. We suggest this strategy could deliver a resource-effective balance of data necessary to plan and calibrate regulator responses and minimise risk to patients of substandard and falsified medicines.
Supporting information
S1 Methods. Detailed methods description for STARmeds study.
https://doi.org/10.1371/journal.pgph.0003999.s001
(PDF)
S1 Table. Market share and weighting of unique study products.
https://doi.org/10.1371/journal.pgph.0003999.s002
(XLSX)
Acknowledgments
STARmeds was a collaboration between Universitas Pancasila, Imperial College London and Erasmus University Rotterdam.
STARmeds study group members are listed alphabetically by institution. Group member roles for this paper are provided in the paper-level supplementary materials.
Universitas Pancasila
Yusi Anggriani, Esti Mulatsari, William Nathanial, Yunita Nugrahani, Elizabeth Pisani, Jenny Pontoan, Ayu Rahmawati, Mawaddati Rahmi, Stanley Saputra, Hesty Utami.
Imperial College London
Adrian Gheorghe, Katharina Hauck, Sarah Njenga, Sara Valente de Almeida.
Erasmus University Rotterdam
Amalia Hasnida.
The team would like to thank Indonesia’s medicine regulator (Badan Pengawas Obat dan Makanan, BPOM) and the national statistics bureau (Badan Pusat Statistik, BPS) for active support in developing the methods described here. The majority of the data collectors were partners of BPS. We thank them for their hard work. We thank BPS statistical consultants Ardi Adji and Budi Santoso.
We also thank members of a multisectoral working group on medicine quality estimation known as PEMO, which groups 12 Indonesian government institutions and 5 professional or industry associations, for advice provided over the course of the study, as well as the members of our Study Advisory Group (Michael Deats, Kharisma Nugroho, Yodi Mahendradhata, Raffaella Ravinetto, Selma Siahaan, Val Snewin, Virginia Wiseman and Firman Witoelar) for helpful advice.
We thank United States Pharmacopeia, who provided reference standards at discounted prices.
References
- 1.
World Health Organization. A study on the public health and socioeconomic impact of substandard and falsified medical products. Geneva: WHO; 2017 Nov. Report No.: WHO/EMP/RHT/2017.02. Available: http://who.int/medicines/regulation/ssffc/publications/Layout-SEstudy-WEB.pdf.
- 2. Pisani E, Rahmawati A. Replication Data for: A randomised survey of the quality of antibiotics and other essential medicines in Indonesia, with sales volume-adjusted estimates of the prevalence of substandard medicines. Harvard Dataverse; 2023.
- 3. Do NT, Bellingham K, Newton PN, Caillet C. The quality of medical products for cardiovascular diseases: a gap in global cardiac care. BMJ Global Health. 2021;6: e006523. pmid:34521627
- 4. Ozawa S, Evans DR, Bessias S, Haynie DG, Yemeke TT, Laing SK, et al. Prevalence and estimated economic burden of substandard and falsified medicines in low- and middle-income countries: a systematic review and meta-analysis. JAMA Network Open. 2018;1: e181662. pmid:30646106
- 5. Zabala GA, Bellingham K, Vidhamaly V, Boupha P, Boutsamay K, Newton PN, et al. Substandard and falsified antibiotics: neglected drivers of antimicrobial resistance? BMJ Global Health. 2022;7: e008587. pmid:35981806
- 6. Saraswati K, Sichanh C, Newton PN, Caillet C. Quality of medical products for diabetes management: a systematic review. BMJ global health. 2019;4: e001636–e001636. pmid:31637025
- 7.
Fleming Fund. Substandard and Falsified Medicines (WHO). In: Fleming Fund [Internet]. Undated [cited 14 Sep 2023]. Available: https://www.flemingfund.org/grants/substandard-and-falsified-medicines-who/.
- 8. Newton PN, Bond KC, Oxford Statement signatories. Global access to quality-assured medical products: the Oxford Statement and call to action. Lancet Glob Health. 2019;7: e1609–e1611. pmid:31708137
- 9.
WHO Expert Committee on Specifications for Pharmaceutical Preparations. Guidelines on the Conduct of Surveys of the Quality of Medicines. WHO Technical Report Series, No. 996, 2016, Annex 7. World Health Organization; 2016. Available: https://digicollections.net/medicinedocs/#d/s22404en.
- 10. Newton PN, Lee SJ, Goodman C, Fernández FM, Yeung S, Phanouvong S, et al. Guidelines for Field Surveys of the Quality of Medicines: A Proposal. PLoS Medicine. 2009;6: e52. pmid:19320538
- 11. STARmeds Study Team. STARmeds study repository: Methods, protocols tools and code. Harvard Dataverse; 2023.
- 12. STARmeds Field Team. Conducting randomised field surveys of medicine quality using mystery shoppers: A practical guide and toolkit. Harvard Dataverse; 2023.
- 13. BPJS Kesehatan. Laporan pengelolaan program 2022 dan laporan keuangan 2022 (auditan). Jakarta, Indonesia: BPJS Kesehatan; 2023. Available: https://bpjs-kesehatan.go.id/#/informasi-publik-detail?slug=laporan-pengelolaan-program-dan-keuangan-bpjs-kesehatan-tahun-2022-auditan.
- 14. Anggriani Y, Ramadaniati HU, Sarnianto P, Pontoan J, Suryawati S. The impact of pharmaceutical policies on medicine procurement pricing in Indonesia under the implementation of Indonesia’s social health insurance system. Value Health Reg Issues. 2020;21: 1–8. pmid:31634791
- 15.
Badan Pusat Statistik. ‘Profil Statistik Kesehatan 2023’. Jakarta: BPS, 20 December 2023. https://www.bps.go.id/id/publication/2023/12/20/feffe5519c812d560bb131ca/profil-statistik-kesehatan-2023.html.
- 16. Lembaga Kebijakan Pengadaan Barang/Jasa Pemerintah. Winners of Indonesian annual public procurement tender, per province, all medicines, 2018–2022. Provided to CEPHAS on request; 2023.
- 17. Statista. Pharmaceuticals—Indonesia. In: Statista [Internet]. Jul 2023 [cited 31 Aug 2023]. Available: https://www.statista.com/outlook/hmo/pharmaceuticals/indonesia.
- 18. Tirtokoesnadi. Industri farmasi perlu perhatian BPJSK, Kemenkes, BPOM, Kemenkeu untuk menghadapi saat Covid-19/virus Corona mewabah dan perkembangan industri farmasi kedepan. GP Farmasi; 2020.
- 19. Ramadhan A. GP Farmasi usulkan alokasi pembayaran obat dalam anggaran BPJS Kesehatan. Antara News. 19 Jan 2019. Available: https://www.antaranews.com/berita/789358/gp-farmasi-usulkan-alokasi-pembayaran-obat-dalam-anggaran-bpjs-kesehatan. Accessed 21 Jan 2019.
- 20. Farmasi GP. Hutang jatuh tempo obat & alkes JKN belum dibayar mencapai Rp. 3,5 T per Juli 2018. 2018.
- 21. Hasnida A, Kok MO, Pisani E. Challenges in maintaining medicine quality while aiming for universal health coverage: a qualitative analysis from Indonesia. BMJ Global Health. 2021;6: e003663. pmid:34049935
- 22.
Reuters. Indonesia finds local trader forged ingredient label in probe of cough syrup deaths. Reuters. 30 Jan 2023. Available: https://www.reuters.com/business/healthcare-pharmaceuticals/indonesia-finds-local-trader-forged-ingredient-label-probe-cough-syrup-deaths-2023-01-30/. Accessed 1 Feb 2023.
- 23. Dewi A, Patel A, Palagyi A, Praveen D, Ihsan BRP, Hariadini AL, et al. A study of the quality of cardiovascular and diabetes medicines in Malang District, Indonesia, using exposure-based sampling. BMJ Global Health. 2022;7: e009762. pmid:36427901
- 24. Slamet L. Policy brief: pengawasan obat dan makanan, termasuk keamanan pangan. Kementrian PPN/Bappenas; 2019.
- 25. Parliament of Indonesia SD. Legislator Nilai Peran BPOM Dalam Pengawasan Obat Bersifat Pasif. 3 Nov 2022 [cited 14 Sep 2023]. Available: http://www.dpr.go.id/berita/detail/id/41503.
- 26.
Kobo Inc. KoBoToolbox | Data Collection Tools for Challenging Environments. In: KoBoToolbox [Internet]. 29 Jun 2022 [cited 29 Jun 2022]. Available: https://www.kobotoolbox.org/.
- 27. Kementerian Kesehatan Republik Indonesia. Farmakope Indonesia Edisi VI. 2020. Available: https://farmalkes.kemkes.go.id/2020/11/farmakope-indonesia-edisi-vi/.
- 28.
United States Pharmacopeia. United States Pharmacopeia National Formulary (43rd edition NF38). USP; 2021.
- 29.
Stata Corp. Stata. Texas, USA: Stata Corp; 2021.
- 30. Gabel J, Lächele M, Sander K, Gnegel G, Sunny-Abarikwu N, Ohazulike RE, et al. Quality of Essential Medicines from Different Sources in Enugu and Anambra, Nigeria. The American Journal of Tropical Medicine and Hygiene. 2024;1. pmid:38740019
- 31.
World Health Organisaton. Survey of the quality of selected antimalarial medicines circulating in six countries of sub-Saharan Africa. Geneva: WHO; 2011. Available: http://www.who.int/medicines/publications/qamsareport/en/.
- 32. Maria V, Tjandrawijaya WN, Rahmawati A, Anggriani Y, Sarnianto P, Pisani E. Are quality medicines affordable? Evidence from a large survey of medicine price and quality in Indonesia. medRxiv; 2024. p. 2024.02.21.24303126.
- 33. Schiavetti B, Wynendaele E, Spiegeleer BD, Mbinze GJ, Kalenda N, Marini R, et al. The Quality of Medicines Used in Children and Supplied by Private Pharmaceutical Wholesalers in Kinshasa, Democratic Republic of Congo: A Prospective Survey. The American Journal of Tropical Medicine and Hygiene. 2018;98: 894–903. pmid:29313479
- 34. Do NT, Boupha P, Newton PN, Caillet C. The quality of antiretroviral medicines: an uncertain problem. BMJ Glob Health. 2023;8: e011423. pmid:36921990
- 35.
World Health Organization. 1 in 10 medical products in developing countries is substandard or falsified. In: World Health Organization [Internet]. 28 Nov 2017 [cited 7 Dec 2021]. Available: https://www.who.int/news/item/28-11-2017-1-in-10-medical-products-in-developing-countries-is-substandard-or-falsified.
- 36. Valente de Almeida, Hauck , Njenga S, Nugrahani Y, Rahmawati A, Rahmi Mi et al. ‘Value for Money of Medicine Sampling and Quality Testing: Evidence from Indonesia’. BMJ Global Health 9, no. 9 (23 September 2024): e015402. pmid:39313254
- 37.
World Health Organizaton. Index of World Pharmacopoeias and Pharmacopoeial authorities; QAS/11.453/Rev.14. WHO; 2022. Available: https://www.who.int/publications/m/item/QAS-11.453-Rev.12.
- 38.
Pisani E. Antimicrobial resistance: What does medicine quality have to do with it? London: Antimicrobial Review; 2015 Nov. Available: http://amr-review.org/sites/default/files/ElizabethPisaniMedicinesQualitypaper.pdf.
- 39. Cavany S, Nanyonga S, Hauk C, Lim C, Tarning J, Sartorius B, et al. The uncertain role of substandard and falsified medicines in the emergence and spread of antimicrobial resistance. Nat Commun. 2023;14: 6153. pmid:37788991
- 40. Murray CJ, Ikuta KS, Sharara F, Swetschinski L, Robles Aguilar G, Gray A, et al. Global burden of bacterial antimicrobial resistance in 2019: a systematic analysis. The Lancet. 2022;399: 629–655. pmid:35065702
- 41.
2021 AWaRe classification. Geneva: World Health Organization; 2021 Sep. Report No.: WHO/HMP/HPS/EML/2021.04. Available: https://www.who.int/publications-detail-redirect/2021-aware-classification.
- 42. Singal GL, Nanda A, Kotwani A. A comparative evaluation of price and quality of some branded versus branded–generic medicines of the same manufacturer in India. Indian J Pharmacol. 2011;43: 131–136. pmid:21572645
- 43. Rahman MS, Yoshida N, Tsuboi H, Sokchamroeun U, Keila T, Sovannarith T, et al. A Cross-Sectional Investigation of the Quality of Selected Medicines for Noncommunicable Diseases in Private Community Drug Outlets in Cambodia during 2011–2013. The American Journal of Tropical Medicine and Hygiene. 2019;101: 1018–1026. pmid:31516106
- 44. Schäfermann S, Wemakor E, Hauk C, Heide L. Quality of medicines in southern Togo: investigation of antibiotics and of medicines for non-communicable diseases from pharmacies and informal vendors. PloS one. 2018;13: e0207911. pmid:30496234
- 45. Hassali MA, Wong ZY, Alrasheedy AA, Saleem F, Mohamad Yahaya AH, Aljadhey H. Perspectives of physicians practicing in low and middle income countries towards generic medicines: A narrative review. Health Policy. 2014;117: 297–310. pmid:25129135
- 46. Patel A, Gauld R, Norris P, Rades T. “This body does not want free medicines”: South African consumer perceptions of drug quality. Health Policy and Planning. 2010;25: 61–69. pmid:19726560
- 47. Riner B, Bussy A, Hélène-Pelage J, Moueza N, Lamy S, Carrère P. “No generics, Doctor!” The perspective of general practitioners in two French regions. BMC Health Serv Res. 2017;17: 707. pmid:29121918
- 48. Dunne SS, Dunne CP. What do people really think of generic medicines? A systematic review and critical appraisal of literature on stakeholder perceptions of generic drugs. BMC Medicine. 2015;13: 173. pmid:26224091
- 49. Pisani E, Dewi A, Palagyi A, Praveen D, Ihsan BRP, Lawuningtyas Hariadini A, et al. Variation in Price of Cardiovascular and Diabetes Medicine in Indonesia, and Relationship with Quality: A Mixed Methods Study in East Java. Am J Trop Med Hyg. 2023;108: 1287–1299. pmid:37160275
- 50. Jayabuana NN. Sistem Lelang JKN Berisiko Gerus Kualitas Obat. 9 Nov 2017. Available: http://industri.bisnis.com/read/20171109/257/707814/sistem-lelang-jkn-berisiko-gerus-kualitas-obat. Accessed 6 Aug 2018.
- 51. Sampurno. BPJS dan E-katalog Obat-obatan. Kompas. 7 Aug 2018. Available: https://www.kompas.id/baca/lain-lain/2018/08/07/bpjs-dan-e-katalog-obat-obatan-2. Accessed 11 Oct 2022.
- 52. Wagnild JM, Lee D, Jayeola B, Lukito PK, Fimbo A, Hampshire K. Can a Smartphone Application Help Address Barriers to Reporting Substandard/Falsified Medical Products? A Pilot Study in Tanzania and Indonesia. Glob Health Sci Pract. 2023; ghsp;GHSP-D-23-00034v1. pmid:37640491
- 53. Ministry of Health Indonesia. Kasus Baru Gangguan Ginjal Akut Pada Anak, Pemerintah Siapkan Langkah Antisipatif. 2023. Available: https://sehatnegeriku.kemkes.go.id/baca/umum/20230206/5642343/kasus-baru-gangguan-ginjal-akut-pada-anak-pemerintah-siapkan-langkah-antisipatif/.
- 54. Vickers S, Bernier M, Zambrzycki S, Fernandez FM, Newton PN, Caillet C. Field detection devices for screening the quality of medicines: a systematic review. BMJ Global Health. 2018;3: e000725. pmid:30233826
- 55. Waffo Tchounga CA, Sacré P-Y, Ravinetto R, Lieberman M, Hamuli Ciza P, Ngono Mballa R, et al. Usefulness of medicine screening tools in the frame of pharmaceutical post-marketing surveillance. PLoS One. 2023;18: e0289865. pmid:37566594
- 56. Ching C, Zaman M h. Impact of ciprofloxacin impurities on bacterial growth, antibiotic resistance development and content assays. Letters in Applied Microbiology. 2021;73: 220–228. pmid:33905563
- 57. Pisani E, Hasnida A, Rahmi M, Kok MO, Harsono S, Anggriani Y. Substandard and Falsified Medicines: Proposed Methods for Case Finding and Sentinel Surveillance. JMIR Public Health and Surveillance. 2021;7: e29309. pmid:34181563