The authors have declared that no competing interests exist.
Prescription sequence symmetry analysis (PSSA), a case-only design introduced in 1996, has been increasingly used to identify unintentional drug effects, and has potential applications as a hypothesis-testing and a hypothesis-generating method, due to its easy application and effective control of time-invariant confounders. The aim of this study is to systematically compare effect estimates from the PSSA to effect estimates from conventional observational parallel group study designs, to assess the validity and constraints of the PSSA study design. We reviewed the MEDLINE, EMBASE, and Web of Science databases until February 2016 to identify studies that compared PSSA to a parallel group design. Data from the eligible articles was extracted and analyzed, including making a Bland-Altman plot and calculating the number of discrepancies between the designs. 63 comparisons (from two studies) were included in the review. There was a significant correlation (p < 0.001) between the effect estimates of the PSSA and the parallel group designs, but the bias indicated by the Bland-Altman plot (0.20) and the percentage of discrepancies (70–80%) showed that this correlation was not accompanied by a considerable similarity of the effect estimates. Overall, the effect estimates of the parallel group designs were higher than those of the PSSA, not necessarily further away from 1, and the parallel group designs also generated more significant signals. However, these results should be approached with caution, as the effect estimates were only retrieved from two separate studies. This review indicates that, even though PSSA has a lot of potential, the effect estimates generated by the PSSA are usually lower than the effect estimates generated by parallel group designs, and PSSA mostly has a lower power than the conventional study designs, but this is based on limited comparisons, and more comparisons are needed to make a proper conclusion.
Conventional observational parallel group studies, such as the cohort study and the case-control study, are still predominantly used to determine causal effects of risk factors and to assess drug safety [
In 1996, Hallas introduced another case-only study design: the prescription sequence symmetry analysis (PSSA) [
Abbreviations: Y = yes, N = no, I = index drug, and M = marker drug.
PSSA has been used less frequently than other, more conventional, pharmacoepidemiologic study designs, and comparisons of PSSA to these designs are lacking [
The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist for this study can be found in the
All identified articles were exported to RefWorks (ProQuest, Michigan). Title and abstract screening were performed and the full text of the relevant studies was reviewed for eligibility by two independent reviewers (D.L.I. and Y.W.). Disagreement between the reviewers was solved by consensus. Studies were eligible for inclusion in the review if they met the following criteria: the article compares (P)SSA to a conventional study design, the data for both study designs comes from the same data source and the definitions for the exposure(s) (index drug/event), outcome(s) (marker drug/event) and risk period(s) are equal for both study designs. Articles were excluded if they were systematic reviews, methodological studies, or studies with simulated data.
For all articles that used (P)SSA as a study design identified by our search, whether the article was eligible for the review or not, the publication year was extracted to examine trends of application of this study design in time. These articles were split up into articles that used PSSA and articles that used another type of SSA, such as event sequence symmetry analysis. If both PSSA and SSA were used, the article was classified according to the principal study design, as identified by the article’s author.
From the eligible articles, we extracted the following data: author(s), year of publication, journal name, type of conventional study design and risk measure, exposure (index drug/event), outcome (marker drug/event), comparator used in the conventional study design, risk period(s), the conventional effect estimate and the PSSA effect estimate. If the study investigated multiple drug pairs, and there was not both a conventional effect estimate and a PSSA effect estimate for all of them, only the data for the drug pairs for which both effect estimates were reported was extracted.
As we compared multiple study designs to each other, rather than using different quality assessment tools for each study design, a method of quality assessment that we employed was to assess the reporting of potential confounders in the eligible articles. We based our assessment on the checklist by Pouwels
The effect estimates were exported to SPSS (IBM, New York, version 23), where they were analyzed using several approaches. First, a scatterplot was made of the effect estimates from the conventional study designs against the effect estimates from the PSSA study design to qualitatively examine potential differences in effect estimates. Second, the Spearman’s correlation coefficients were determined to evaluate the correlation between the effect estimates. Third, we examined whether the different study designs found the same significant associations. Fourth, because correlation may not be ideal to measure agreement between two types of study design, a Bland-Altman plot was made to assess this [
The search identified 183 unique articles. Based on the title and abstract screening, 85 potentially relevant articles were selected for full-text screening. After reviewing for eligibility, two articles were included into the review (
Author, year | Journal | Number of drug pairs | Conventional design and effect estimate | Exposure | Comparator in the conventional design | Outcome | Risk period |
---|---|---|---|---|---|---|---|
Corrao, 2005 [ |
Pharmacoepidemiol. Drug Saf. | 62 | Cohort (SIR) and nested case-control (AOR) | Incident use of antibacterials | No incident use of antibacterials | Arrhythmia (incident anti-arrhythmic prescription) | 18 months |
Garrison, 2012 [ |
Arch. Intern. Med. | 1 | Cohort (HR) | Incident use of inhaled long-acting β2-agonists | Incident use of anticholinergics | Nocturnal muscle cramps (incident quinine prescription) | 12 months |
Abbreviations: SIR, standardized incidence ratio; AOR, adjusted odds ratio, HR, hazard ratio.
There were 50 articles (
Number of PSSA articles (black bar) and number of SSA articles (grey bar) published per year and the total number of articles indexed in MEDLINE (black line) per year after the introduction of the method in 1996 (first bar in the graph).
A scatterplot was made of the conventional effect estimates against the PSSA effect estimates (
Spearman Rank-Order Correlation tests were performed (
Dataset | N | Spearman’s correlation coefficient | |
---|---|---|---|
All results | 63 | 0.621 | 5.521·10−8 |
PSSA vs. cohort | 35 | 0.553 | 0.001 |
PSSA vs. nested case-control | 28 | 0.676 | 7.900·10−5 |
Besides measuring correlation, a second approach used to compare the two methods was to assess if the different study designs found the same significant associations between the index- and marker drugs. In the first article, there were two significant signals found by all three methods, two signals that were only found with the conventional study designs and two additional signals that were identified only with the cohort design. In the second article, all designs found a statistically significant association for the investigated drug pair. So, combining data from both articles, the PSSA method identified less significant, potentially causal, associations between the index- and the marker drug than the two parallel group designs.
The third approach used to compare the effect estimates from the different study designs was to make a Bland-Altman plot to assess the degree of agreement between them (
The black dots represent the difference against the mean of the effect estimate pairs; the black continuous line is the mean difference between the effect estimates (with 95% CI represented by the black dotted lines); the grey dashed lines represent the upper- and lower limit of agreement (with 95% CI’s represented by the grey dotted lines).
A fourth approach to compare the effect estimates is by determining discrepancies between them, as described before. The number and percentage of discrepancies are shown in
Dataset | Number (%) of discrepancies |
Number (%) of discrepancies for which the conventional effect estimate is larger |
Number (%) of discrepancies for which the conventional effect estimate is further away from 1 |
---|---|---|---|
All results (n = 63) | 46 (73) | 43 (94) | 16 (35) |
PSSA vs. cohort (n = 35) | 24 (69) | 22 (92) | 8 (33) |
PSSA vs. nested case-control (n = 28) | 22 (79) | 21 (96) | 8 (36) |
aDiscrepancies were characterized by the natural logarithm of the PSSA effect estimate being ≥50% larger or smaller than the natural logarithm of the conventional study design effect estimate.
bThe fraction of the total number of discrepancies for which the conventional effect estimate is larger than the PSSA effect estimate.
cThe fraction of the total number of discrepancies for which the conventional effect estimate is further away from 1 than the PSSA effect estimate.
This study aimed to assess and quantify the correlation, agreement, and discrepancies between effect estimates from the PSSA and two conventional pharmacoepidemiologic study designs, the cohort- and nested case-control study design. We found that there was a significant correlation between the effect estimates of the PSSA and effect estimates of the conventional studies, but this strong correlation was not accompanied by similar effect estimates; there were systematic differences between the effect estimates generated by the two types of design. The Bland-Altman analysis showed significant bias between the effect estimates, with the effect estimates from the conventional study designs being, on average, 0.20 higher than the effect estimates from the PSSA.
The difference in effect size between the two types of design could originate from the use of a reference group in the conventional study designs, while the PSSA is a case-only design. Time-invariant confounders, whether registered or unregistered, such as advanced age, female gender, and hypochondriasis, may result in bias in parallel group designs (if they are not adjusted for) but not in the case-only PSSA [
However, this may not be the reason for the discrepancies if important assumptions for the validity of PSSA are not met. The assumptions of PSSA, based on the strengths and limitations of the method, are: there is an appropriate and specific indicator/proxy for the outcome, the proxy can be prescribed independently of the sequence of the exposure to the index drug and the occurrence of the outcome (e.g. if the outcome is fatal, the proxy could only be prescribed after incident index drug use), the outcome of interest has no effect on subsequent treatment, the effect of the exposure is transient, and the drug-induced symptom is relatively unknown to the prescribing physician [
The included articles used relatively long risk periods; 12- and 18 months. These are quite wide time-windows since exposure to the index drug shortly before the onset of the adverse event is more likely to be causal for the investigated exposures, especially for the antibacterials exposures from the study by Corrao
Additionally, the underestimation of the effect size by the PSSA compared to the conventional study designs could be caused by the use of the relatively long risk periods in the PSSA. Using a longer risk period than necessary, especially in the case of the antibacterials exposure, could have diluted the signal by including more nonspecific sequences in the calculation of the adjusted sequence ratio. If the use of a longer risk period has more effect on the PSSA study design than on the other two designs, this could account (in part) for the lower effect estimates generated in this study design.
More differences were observed when assessing the number of adverse event signals (i.e. statistically significant results). There were cases where adverse event signals were only measured in the cohort- and nested case-control designs, but not in the PSSA. Since PSSA is only performed on subjects that filled incident prescriptions for both the index- and the marker drug, the sample size of the PSSA is smaller, and therefore the power lower, compared to that of the conventional designs. A possible solution for this was used by Pratt
In general, for all arguments made above, it must be noted that our results are only based on two separate studies, and that it is therefore not possible to generalize these results to all studies that use PSSA and/or conventional study designs. A large part of our results (62 of the 63 comparisons included, 98%) are derived from a single study, and therefore these comparisons cannot be considered independent, and the results of our review are greatly dependent on the methodology and results used in that particular study. It may be the case that some part of the methodology of this study caused a systematic underestimation of the effect size of the PSSA study design compared to a conventional study design, and that application of the PSSA methodology is the cause of the discrepancy rather than the PSSA methodology itself. To be able to draw any generalizable conclusion from comparisons between the PSSA study design and conventional study designs, more independent comparisons between the two are greatly needed.
To assess the reporting of potential confounding in the articles used for this review, we used the checklist by Pouwels et al, derived from the “Strengthening the Reporting of Observational Studies in Epidemiology” (STROBE) statement [
One important advantage of PSSA is that it eliminates time-invariant confounders, but PSSA may still be sensitive to other types of confounding. Both articles calculated the adjusted sequence ratio to eliminate confounding by temporal prescribing trends and one article also adjusted for age and seasonal trends in drug prescription. Contra-indication was identified as a potential confounder, but this confounder is hard to control. A large part of the results of this review originates from a study that aimed to determine the effect of antibacterials on arrhythmia. The study split up the antibacterials into many subgroups and this division could have introduced confounding by contra-indication: physicians might only prescribe certain antibacterials to patients who are known to be at risk of arrhythmia. While this bias would not be the cause of the difference between the study designs, because this same division was made for the conventional study designs, it is an important confounder to take into consideration when using PSSA to compare subgroups of drugs prescribed for a similar indication. This is especially true when prescribing physicians are already aware of the potential adverse event, and may let this influence their prescribing behavior in choosing a specific drug for high-risk patients.
Also, when subjects consult their physician when they present with symptoms of an adverse event, some physicians may discontinue the index drug instead of prescribing the marker drug. In this case, these subjects would be missed by the PSSA, resulting in an underestimation of the effect estimate. Furthermore, it would also reduce the sample size, which further decreases the power of PSSA. However, most of the effect estimates from the conventional designs were also based on drug dispensing data, so this would not cause discrepancies between the effect estimates of both types of design studied in this review. Besides, it was demonstrated that even fairly well known adverse events are often treated by prescribing additional drugs, rather than discontinuing the drug that might have caused it [
This is the first systematic review that assesses the performance of PSSA compared to conventional study designs by reporting on the agreement and the discrepancies between the effect estimates and the direction of the discrepancies. A challenge that we have faced was the limited data available since empirical comparisons between PSSA and conventional designs are rare, and therefore it is hard to draw any definite conclusions from our results. We recognize that this could be due to the fact that researchers may perform both types of study and then publish the study with the most relevant results. This would underestimate the discrepancies between the study designs. Another possibility is that the results for both types of study were so similar that researchers only chose to publish the results of one of the two study designs. This would result in an overestimation of the difference between the study designs. Also, because the study designs use different effect measures, they may not be readily comparable, causing inconsistencies that are based on the different effect measures used rather than the different study designs. Therefore, there is a need for a quantitative statistical test that compares effect estimates from different risk measures.
PSSA, due to its simple and quick implementation and its ability to eliminate time-invariant confounding has a lot of potential in assessing drug safety. However, our results indicate that PSSA lacks power in many situations, and its results often significantly deviate from effect estimates generated by conventional parallel group study designs. PSSA might, therefore, be more suitable as a hypothesis-generating design, that should be followed by a more conventional parallel group design for hypothesis-testing and confirmation. Our results should be approached with some caution, though, as they are only based on two independent studies. To get a better understanding of the practical differences between the two types of designs, and to be able to make any generalizations, more comparisons between PSSA and parallel group designs are required. Future studies should also compare PSSA to randomized controlled trials, to assess how PSSA performs against the study design considered to be the golden standard.
(PDF)
All articles that addressed (P)SSA as a study design are included in this list, regardless of whether they were eligible to be included in the systematic review.
(PDF)
Grey diamonds (with continuous grey line): PSSA vs. cohort with accompanying trend line; black squares (with continuous black line): PSSA vs. nested case-control with accompanying trend line. The dashed grey line represents the line y = x. Abbreviations: SIR, standardized incidence ratio; HR, hazard ratio; AOR, adjusted odds ratio; ASR, adjusted sequence ratio.
(PDF)