Introduction

Oncology-associated adverse drug/device reactions are serious medical toxicities. These toxicities encompass three areas. Oncology drugs are the most toxic pharmaceutical class [1,2]. Oncology patients are ill, with comorbid illnesses, extensive cancer, and undergoing toxic treatments. When adverse reactions occur that represent severe oncology-drug induced gastrointestinal toxicity, arterial or venous thromboembolism, cardiac arrhythmias, opportunistic infections, or hemorrhage, these events are frequently attributed to the cancer, comorbid illness, or the “cost of doing business.” [3–6] A second toxicity type relates to oncology-drug induced growth or dissemination of existing cancers- as with erythropoiesis stimulating agents or morcellator procedures.[7,8] A third type relates to new cancer development as postulated with natalizumab associated melanoma or protein pump inhibitors associated gastric malignancies.[9,10] Two National Institutes Health R01 funded pharmacovigilance programs (the Southern Network on Adverse Reactions (SONAR) and the Research on Adverse Drug Events and Reports (RADAR), projects), conducted beginning in 1998, have collaborated with clinicians to report > 50 serious adverse drug/device reactions from the oncology setting.[3,6] These co-investigators are aware of efforts of academic collaborators who also reported similar serious oncology-associated adverse drug reactions. SONAR and RADAR co-investigators and colleagues have personally treated oncology patients who developed several life-threatening adverse drug reactions, evaluated these patients, and published on the cause and cases of these toxicities. These investigations were frequently followed by label warnings and/or convening of Food and Drug Administration (FDA) Advisory Committee reviews of safety findings. Clinicians who take “extra-steps” of evaluating serious adverse drug reactions and reporting this in medical publications may be subject to repercussions or alternatively, career enhancements. Our purpose is to shed light on motivations and experiences of clinicians who worked with investigator leaders of two National Institutes supported pharmacovigilance networks and who also personally evaluated and reported unrecognized serious adverse drug reactions.

Materials and methods

Our study is a qualitative analysis of experiences who collaborate with leaders of two NIH funded pharmaceutical safety grants and subsequently published case series of serious oncology-associated (adverse drug reactions). The ADRs were initially seen as an individual case by the interviewed clinician. We used “The Criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups” to guide the design, analysis, and reporting of our study.[11] As recommended by the COREQ guidelines, the qualitative researchers in this study varied in credentials, training, occupation, and training. The principal investigator was a qualitative analyst PhD researcher. A senior co-investigator/interviewer was a MD PhD health services researcher/policy endowed chair professor who had previously published physicians reporting of health care fraud, and another co-investigator was a JD PhD health policy tenured professor who had conducted qualitative and quantitative research, including one manuscript on physicians reporting of health care fraud. The theoretical framework was based on an exploratory approach to support hypothesis generation- relevant considerations since there are no published data about clinician experiences with identifying and reporting case series of serious adverse-drug reactions. The analytic focus was based on quantitative analysis and systematic organization of data into a structured format.

No repeat interviews were conducted. After 14 interviews were conducted and the responses analyzed, the Principal Investigator, the co-Principal Investigator, and the study team jointly determined that saturation had occurred. Transcripts were not returned to the interviewees, but were reviewed by the co-Principal Investigator. Data coding and transcription were done by four research assistants, with two research assistants independently coding and transcribing each interview. The co-Principal Investigator provided to the research assistants the agreed upon coding terms that guided transcript analysis. Themes were derived in advance, based in part on personal experiences of the co-Principal Investigator treating persons with serious oncology-associated adverse drug reactions and subsequently reporting this information in the medical literature. Since the interview transcript size was small (about five pages per interview), no software analysis package was used. Participants reviewed the draft manuscript (prior to submission) and were asked to provide relevant feedback, particularly if each interviewee did not express concerns about linking of responses to individual interviewees. Quotations were presented to illustrate each of the selected themes. Participant numbers are included in the quotations. The data and the findings exhibited strong consistency, as assessed by the study investigators. The major themes are presented in each table and in the corresponding text. Minor themes are also highlighted in the tables and the text. Overall, the research team, study design, and analysis/findings were consistent with the 32 consolidated criteria for reporting qualitative studies (COREQ).[11] The details of the process are shown in a flow chart in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Flow chart for study inclusion criteria.

https://doi.org/10.1371/journal.pone.0219521.g001

The qualitative approach was chosen as it is a rigorous approach for investigating personal motivations, reflections, and outcomes among a small group of persons who have a common experience.[12] Qualitative methods are also considered appropriate for studies that provider exploratory research and hypothesis generation. These are appropriate considerations as there are no prior research publications that report on clinician motivations for publishing serious adverse drug reaction information. The study received approval from the University of South Carolina Institutional Review Board. All participants provided verbal consent.

The operational definition of serious adverse drug reactions were events attributed by the reporting clinician as causing death or potentially fatal organ toxicity. Publication dates are redacted to ensure clinician anonymity. Most reports were followed by boxed warnings and/or FDA Advisory Committees. The study interviewer was an MD PhD MPP interviewer whose research focused on pharmaceutical safety research and impacts of this work on academic clinicians who report serious adverse drug reactions. This interviewer was trained by an experienced qualitative research journalism professor who also conducted the first interview as part of the training session. The interview guide was developed by the experienced qualitative researcher. (S1 Interview Guide) The guide underwent field review by three additional PhD health services researchers with expertise in drug safety. The guide contained a list of key thematic areas to be probed for clarification and additional detail for each of four pre-selected topic areas- interviewee views of responses by employees of the pharmaceutical manufacturers, employees of the FDA, and clinicians in academic medical centers and recommendations that the interviewee considered as relevant for future pharmacovigilance efforts. All interviews were recorded and transcribed. No interviewee requested a copy of the interview guide prior to the interview. The interviewer also kept written notes. The interview began by obtaining verbal informed consent, followed by demographic and professional information on the interviewee. Next, we elicited information on the nature of the work that led to identification of the index case of the serious adverse drug reaction and to their follow-on work to establish a case series. Interviews then proceeded chronologically through the following topics: the interviewees personal assessment of responses to the study findings by employees of the relevant pharmaceutical manufacturer, safety personnel of the FDA, clinicians in academia at either the home institution of the interviewee or an academic institution where the interviewee had applied for a possible academic position, and recommendations by the interviewee that might be considered when future studies evaluate potential serious adverse drug reactions. Interviewees were asked to verify that they had not participated in any “whistle blower” investigation and had not received any income for publishing their work.

We analyzed interview transcripts using standard coding techniques and the constant comparative methods of qualitative analysis, based on methods reported previously.[13,14,15] The two senior investigators independently analyzed the interviews and developed separate coding schemes for organization the data. Coding schemes were then compared, discussed, and reconciled to product a final coding structure which consisted for six broad themes covering 20 specific codes. One investigator (CLB) then used the final coding scheme to code all of the transcripts and interview notes, with input from each of the four transcribers. The transcripts and interview notes were coded using manual counts for each of the 20 specific codes. The analysis focused on organizing and describing themes, including distinctions among clinicians based on their academic training (clinician/PhD versus clinician) and level of training (fellow or student versus assistant or associate professor).

Results

We identified 18 eligible persons for interview. Three interviewed persons were excluded as they did not have first-hand information on index cases. One clinician who was working at a pharmaceutical corporation did not return telephone calls. We conducted semi-structured 30-minute telephone interviews at the workplace with persons who we identified as clinicians who were collaborators with a key investigator of SONAR and RADAR projects or who collaborated closely with a senior RADAR/SONAR co-investigator, had evaluated a case of a serious oncology-associated adverse drug reaction, personally conducted research to identify cases of persons with this toxicity, and then published the findings in peer-reviewed medical journals or, in one case, presented findings to FDA personnel. The study sample was felt to be exhaustive of all individuals who met the entry criteria. All but one of the 15 eligible participants were interviewed- the lone exception had left academic medicine and could not be contacted. The contacted clinician received an email querying if there would be interest, then a phone call from the study co-Principal Investigator confirming interest in participation, and then was emailed a study description that included a consent form. A telephone interview was conducted with the study participants with a research assistant taking notes and taking the conversation and the co-Principal Investigator and the interviewee speaking for about 30 minutes. Three potential participants were not involved in clinical medicine. The characteristics of the interviewed persons are outlined in [Table 1]. The interview guide was developed by a journalism PhD collaborator with extensive expertise in qualitative research and telephone-based interview studies. Interviews lasted a median of 35.5 minutes (interquartile range, 30 to 40 minutes) addressed clinician motivation and experiences. All of the 14 included transcripts were found to have > 98% concordance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Table of adverse drug reactions.

https://doi.org/10.1371/journal.pone.0219521.t001

Overall, data were included from 12 clinicians who had personally treated a patient who developed a serious and previously unreported serious adverse drug reaction, one clinician whose wife died from a previously unrecognized adverse device reaction, and one MD PhD student whose mentor had treated the index patient. Five clinicians had MD/PhD or DDS/PhD degrees, one clinician was enrolled in an MD/PhD program, and six clinicians had extensive basic and/or clinical science training. The sample included six medical oncologists, one radiation oncologist, one oncology PharmD, five surgeons, and one MD PhD student who worked closely with an oncologist oncology mentor. Studies were initiated following review of a single case (n = 3) or after treating between four and 65 cases with the relevant toxicity (n = 11), or after reviewing clinical trial data involving for 350 and 800 patients in two phase III clinical trials (n = 2). Six reports were single-site case series, two were phase II single-site clinical trial reports, one was a meta-analysis, two were phase III clinical trial reports, and three were observational studies from a small number of academic medical centers.

Investigators included one medical student, three oncology fellows, and 10 clinicians who were assistant/associate professors at academic medical centers when the case series was evaluated. Adverse drug reactions included fatal infections of the central nervous system (n = 1), bone osteonecrosis (n = 3), tumor progression/development (n = 5), arterial or venous thromboembolism (n = 3), cardiac arrythmias (n = 1), or fatal gastrointestinal toxicity (n = 1). Five studies were disseminated in four high-impact factor medical journals (JAMA, Lancet, Annals of Internal Medicine, and New England Journal of Medicine), five studies were disseminated in two high-impact factor hematology-oncology journals (Blood and the Journal of Clinical Oncology), two studies were disseminated in two surgical specialty journals, one study was disseminated in a lower impact-factor oncology journal, and one study was disseminated by the Government Accountability Organization.[8]

Following dissemination of safety findings for eleven and seven studies, respectively, boxed warnings were added to product labels and FDA advisory committees were convened. Related peer-reviewed publications were cited in medical publications a median of 1,114 times (range 290 to 2,500 for 13 reports) or in national newspapers (for 7 reports). The median estimated incidence of the adverse events was 1 in 6 treated patients (range 1 in 3 to 1 in 10,000 (13 reports)). Five described toxicities that were fatal. Following seven adverse drug reaction reports, phase III clinical trials were temporarily discontinued while additional safety evaluations were conducted. For one toxicity, a manufacturer voluntarily discontinued manufacturing the implicated device. Seven study reports were submitted to the FDA for additional review, and 10 investigators submitted related findings to pharmaceutical manufacturers. [Table 1]

Motivations

All but one of the clinicians intended to publish the findings, generally aiming for publication in a top-tier medical journal or medical/surgical specialty journal. One clinician retained attorneys who filed a device liability lawsuit against a manufacturer and submitted safety data to the FDA, but did not submit safety data to a peer-reviewed journal. Reported motivations coalesced around five non-mutually exclusive themes: scientific curiosity, public health, a desire to inform the broad medical community of a toxicity that had not been reported previously, ethics, and because the toxicity had affected a clinician’s wife.

The most common theme, scientific curiosity, was linked by six clinicians to their training as a clinician scientist. One clinician reported that “the MD side of me identified the toxicity and the PhD side of me facilitated my scientific investigation of the causes of the toxicity”. Another clinician reported that “as a MD PhD investigator, this is the kind of science that I have been trained to do”. Three clinicians reported that training as clinician scientists led to the investigation. Five of six clinicians who reported scientific curiosity as the main motivator were investigators of National Cancer Institute-sponsored or pharmaceutical manufacturer funded investigator-initiated trials.

A less common theme was public health related, with three clinicians reporting that toxicity findings had public health implications that they felt was their responsibility to report. One clinician said “what got me personally was seeing the denials of this information and the callous disregard [of the pharmaceutical manufacturer] for this complication”. A third theme, ethics, was endorsed by two clinicians. One clinician cited the Belmont Report as a motivating factor. One clinician, endorsing personal reasons for toxicity evaluation, stated that “I have to tell you without a question if this problem had not happened to us [the clinician’s spouse died from the toxicity], I would have probably agreed that this was a problem but I wouldn’t have done anything more than that. As many of my colleagues did”. [Table 2]

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Primary motivations for publishing adverse drug reactions.

https://doi.org/10.1371/journal.pone.0219521.t002

Responses by the academic community

The academic community was reported as being supportive for six clinicians, unfavorable towards four clinicians, and neutral towards two clinicians. Three clinicians reported that following the publishing of their case series, they were invited to speak at academic institutions worldwide. One clinician reported “Well, the chairman of my department said this is really significant…I say, well, mostly, he was right”. In contrast, two clinicians reported that during job interviews, a senior academic interviewer indicated that reporting of the serious adverse drug reactions would not be an academic pursuit that should be continued. One clinician stated “the chairman where I interviewed said you will not make friends with this kind of research”. Two clinicians were not comfortable sharing information about responses that they had received from the medical community. [Table 3]

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Interviewee report of academic community response.

https://doi.org/10.1371/journal.pone.0219521.t003

Reponses by the pharmaceutical community

The pharmaceutical community was generally negative towards the clinicians, although two clinicians reported collaborative responses. One clinician said that his wife called the post-reporting time the “Maalox month”–primarily in response to intimidation by a physician employed by a pharmaceutical manufacturer. Another clinician met with the Chief Executive Officer of the relevant pharmaceutical manufacturer who said he wanted to see “in person the [expletive] who had cost us $2 billion”. A third clinician was sued by the pharmaceutical device manufacturer. Safety findings were subsequently central aspect of legal settlements paid by a manufacturer to the Department of Justice of $600 million (two clinicians) and $300 million (two clinicians), although these clinicians did not receive financial rewards. One clinician reported that the manufacturer collaboratively offered to identify additional cases of the relevant toxicity that were in the manufacturer’s safety database. One clinician reported that the relevant manufacturer was fairly negative, while a competitive manufacturer placed this person on an academic advisory panel and sponsored the clinician to meet with safety representatives of the FDA. [Table 4]

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Interviewee report of pharmaceutical company response.

https://doi.org/10.1371/journal.pone.0219521.t004

Responses by the FDA

FDA responses were characterized as collaborative for two clinicians, neutral for five clinicians, unresponsive for two clinicians, and negative for six clinicians. One of these instances was described as collaborative and negative. Some persons at the FDA worked with the clinician to facilitate market withdrawal of the implicated device. Other FDA persons were reported as being unresponsive although other clinicians had previously reported device-associated deaths to the FDA. One clinician reported that FDA safety officials were not responsive because he/she “practiced in a foreign country, and that FDA safety officials appeared to be more responsive to United States’ clinicians”. One clinician reported that the response was vigorous and positive- the FDA convened an independent auditing panel to review the safety findings. One clinician who had submitted >60 reports to FDA’s MedWatch database was surprised when no follow-up from FDA personnel occurred. [Table 5]

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Interviewee report of fda employee response.

https://doi.org/10.1371/journal.pone.0219521.t005

Discussion

This study of 14 clinicians who have published reports of, or potentially fatal oncology associated adverse drug/device reactions found that responses of the relevant pharmaceutical manufacturer were primarily negative, responses by fellow academic clinicians were generally supportive or neutral, and responses by FDA personnel were supportive in only two instances. The clinical findings were significant, with many reports being published in high-impact medical journals, being described in revised product labels and in “Dear Doctor” letters, and in four instances, being associated with settlements by pharmaceutical manufacturers for > $900 million. In interpreting our findings, several factors should be considered.

The most common motivation was scientific curiosity. This was reported by three of the five PhD clinicians as well as the MD/PhD student who indicated that the physician background facilitated recognition of the toxicity and the PhD background facilitated detailed basic science investigation. In one instance, a clinician reported the toxicity after his wife had developed device-associated metastatic sarcoma (and ultimately died from this toxicity). There are no prior studies in the published literature that report on motivations of clinicians to publish their investigations of adverse drug reactions that they have observed in clinical practice. Kesselheim et al. reported that whistle-blowers who report on possible fraud by their employers, including four persons who reported that safety data had not been appropriately disseminated, focused on integrity, altruism or public safety, justice, and self-preservation as motivating factors.[13] Formal clinical trial training resulted in one clinician invoking principles of the Belmont Report as part of the impetus for pursuing investigation of potential causes of these serious adverse drug reactions.

Twelve clinicians reported that they had received negative feedback personally or professionally. This too mirrors findings reported for pharmaceutical whistleblowers.[13] Similar experiences have been reported by clinicians who reported rofecoxib-associated cardiac mortality, gadolinium-associated nephrogenic systemic fibrosis, and even aspirin-associated Reye’s syndrome.[16] Kesselheim suggested that for pharmaceutical whistleblowers, ensuring responsible “whistle-blowing” would require interventions that strengthen penalties against retaliation.[11] Similarly, for clinicians, protection against retaliations from pharmaceutical manufacturers are needed. In some instances, this might include financial support if lawsuits against the clinician are filed in an effort to derail the scientific investigation of serious toxicity.

It should be noted that reporting safety findings to the FDA was not a route chosen by 12 clinicians. One clinician who submitted > 60 individual adverse event reports to FDA’s MedWatch program did not receive any feedback from the FDA. While only 1% to 10% of all serious adverse drug reactions are estimated to be reported to the FDA, it is undoubtedly even more rare for a clinician to undertake a detailed investigation of safety data for a large series of patient with the suspected toxicity and then report this information in peer-reviewed medical publications [17]. In one instance, the reporting clinician received a R01 grant from the National Institutes of Health to conduct basic science studies of the toxicity.

In three personal stories we heard, feedback from fellow academics was negative. As suggested by some policy makers, physicians should be able to anonymously report their safety findings to regulatory authorities. This anonymity is not possible with peer-reviewed publications. Extensive influence of pharmaceutical manufacturers on academic communities is concerning, as noted in the high rate of physicians with extensive pharmaceutical funding who participate in FDA Advisory Committee meetings and on medical society-sponsored clinical guideline panels. This influence undoubtedly spills over into academic and FDA environments.

Fourth, personal experience of the clinician interviewees to recommendations for improving pharmaceutical safety are informative. The FDA endorses some of these recommendations with its support of the SENTINEL big-data initiative, which has yielded important safety findings.[18] An important recommendation is for the FDA or the National Institutes of Health to actively support pharmaceutical safety centers of excellence. The Centers for Education and Research of Therapeutics (CERTs) program that was recently disbanded might be a strong candidate for reinstating.[19] Consistent with existing qualitative research studies on barriers to reporting adverse drug reactions, these initiatives do not require additional clinician time or do not result in additional clinician burden.

This study has limitations. We focused on published case series of severe adverse drug reactions that had been investigated by SONAR or RADAR investigators or close collaborators of RADAR or SONAR investigators. These investigators were located at > 50 universities, hospitals, safety organization, or Veterans Administration medical centers. Interview materials obtained from these individuals included highly sensitive personal information that would be difficult to obtain from other clinicians. Our findings therefore may not be generalizable to other clinicians who evaluate serious adverse drug reactions and publish these findings in the peer-reviewed medical literature. Responses to queries about motivation to pursue the scientific investigation may reflect a socially desirable response bias. Finally, recall bias may affect interviewees’ responses.

Conclusions

Our study provides some of the first reports of clinician motivations and experiences with reporting serious or potentially fatal oncology-associated adverse drug or device reactions. Overall, it appears that negative feedback from pharmaceutical manufacturers and mixed feedback from the academic community and/or the FDA were identified by clinicians who participated in this study. “Big data,” registries, Data Safety Monitoring Boards, and pharmacogenetic studies may facilitate improved pharmacovigilance efforts for oncology-associated adverse drug reactions.[20,21] These initiatives overcome concerns related to complacency, indifference, ignorance, and system-level problems as barriers to documenting and reporting adverse drug events- barriers that have been previously reported for clinician reporting of serious adverse drug reactions. [16,17,19]

Supporting information

Acknowledgments

We are particularly indebted to the long series of conversations and guidance on this project from the late Robert C Kane MD, the 2016 Frances Kelsey Award Recipient from the Food and Drug Administration who died in 2018 before this manuscript had been completed. We would like to thank clinicians and pharmacovigilance experts who have worked closely with the National Institutes of Health-funded R01 grants, the Southern Network on Adverse Reactions (SONAR) and the Research on Adverse Drug Events and Reports (RADAR) projects from 1998 to the current day at: the University of South Carolina, Columbia, South Carolina; the Medical University of South Carolina, Charleston, South Carolina; Duke University, Durham, North Carolina, University of North Carolina, Chapel Hill North Carolina, Tulane University and the Louisiana State University, New Orleans, Louisiana, the University of Miami, the University of Texas, San Antonio, Texas; the University of New Mexico, Albuquerque, New Mexico; Nara Medical University, Kashihira, Nara, Japan; the Japan Institute of Pharmacovigilance, Tennoji-ku, Osaka, Japan, the MD Anderson Cancer Center, Houston, Texas; Johns Hopkins University, Baltimore, Maryland; the University of Nebraska School of Medicine, Omaha, Nebraska; Baylor University, Houston, Texas; the University of Chicago, Chicago, Illinois’ the University of Illinois/Chicago, Chicago, Illinois; Indiana University, Indianapolis, Indiana; Case Western Reserve University, Cleveland, Ohio; Albert Einstein College of Medicine, Bronx, New York; the Mission Saint Joseph’s Hospital, Asheville, North Carolina; University of Copenhagen, Copenhagen, Denmark; Yale University, New Haven, Connecticut; University College, London, England; University of Modena and Regio Emillia, Modena, Italy; the Eunice Kennedy Shriver National Institute of Child Health and Human Development, the National Institutes of Health, Bethesda, Maryland; the Food and Drug Administration; Silver Spring, Maryland; the University of Copenhagen, Copenhagen, Denmark; McGill University, Montreal, Canada; the University of Pennsylvania, Philadelphia, Pennsylvania; Rutgers University, Piscataway, New Jersey; Washington University, Saint Louis, Missouri; Memorial Sloan Kettering Cancer Center, New York, New York; the Midwest Center for Health Services Research and Policy Studies, Maywood, Illinois; Tenon Hospital and Pierre and Marie Curie University, Paris France; Singapore General Hospital, Singapore; Hotel-Dieu, Paris, France; Queen’s University, Kingston, Ontario; the University of California at Los Angeles, Los Angeles, California; A Manzoni Hospital, Lecco, Italy; the University of Utah, Salt Lake City, Utah; Oregon Health Sciences University, Portland, Oregon; the Therapeutics Good Administration, Cabrera, Australia; Long Island Jewish Medical Center, New Hyde Park, New York; the Veterans Administration Cooperative Studies Program Clinical Research Pharmacy, Albuquerque, New Mexico; University of Pisa, Pisa, Italy; Stanford University, Palo Alto, California; the University of Pisa, Pisa, Italy; the University of Melbourne, East Melbourne, Australia; the Dana Farber Cancer Institute, Boston, Massachusetts; the Neoplastic and autoimmune diseases research institute, Rancho Santa Fe, New Mexico; the Southern Illinois College of Medicine, Springfield, Illinois. The names of the individual investigators are not included to protect the confidentiality and anonymity of collaborators of RADAR and SONAR. We are particularly indebted to the 17 individuals from these institutions who gave oral consent to participate in the project (that was heard by two interviewers). We would like to acknowledge the contributions of Shannon Bowen PhD who developed the study interview script and also provided training for the study interviewer. We would also like to acknowledge the contributions of Paul S Ray DO and Shamia Hoque PhD with revisions of the manuscript.

Presentations: Parts of this paper have been presented at the American Medical Association 16^th Annual Research Symposium in National Harbor, Maryland on November 9, 2018 and at the American Society of Clinical Oncology Annual Conference in Chicago, Illinois on June 1, 2019.