The Use of Google Trends in Health Care Research: A Systematic Review

Background Google Trends is a novel, freely accessible tool that allows users to interact with Internet search data, which may provide deep insights into population behavior and health-related phenomena. However, there is limited knowledge about its potential uses and limitations. We therefore systematically reviewed health care literature using Google Trends to classify articles by topic and study aim; evaluate the methodology and validation of the tool; and address limitations for its use in research. Methods and Findings PRISMA guidelines were followed. Two independent reviewers systematically identified studies utilizing Google Trends for health care research from MEDLINE and PubMed. Seventy studies met our inclusion criteria. Google Trends publications increased seven-fold from 2009 to 2013. Studies were classified into four topic domains: infectious disease (27% of articles), mental health and substance use (24%), other non-communicable diseases (16%), and general population behavior (33%). By use, 27% of articles utilized Google Trends for casual inference, 39% for description, and 34% for surveillance. Among surveillance studies, 92% were validated against a reference standard data source, and 80% of studies using correlation had a correlation statistic ≥0.70. Overall, 67% of articles provided a rationale for their search input. However, only 7% of articles were reproducible based on complete documentation of search strategy. We present a checklist to facilitate appropriate methodological documentation for future studies. A limitation of the study is the challenge of classifying heterogeneous studies utilizing a novel data source. Conclusion Google Trends is being used to study health phenomena in a variety of topic domains in myriad ways. However, poor documentation of methods precludes the reproducibility of the findings. Such documentation would enable other researchers to determine the consistency of results provided by Google Trends for a well-specified query over time. Furthermore, greater transparency can improve its reliability as a research tool.


Introduction
New tools are emerging to facilitate health care research in the Big Data era. One form of Big Data is that which accumulates in the course of Internet search activities. Internet search data may provide valuable insights into patterns of disease and population behavior. [1] In fact, the Institute of Medicine recognizes that the application of Internet data in health care research holds promise and may ''complement and extend the data foundations that presently exist''. [2] An early and well-known example of utilizing Internet data in health has been the surveillance of influenza outbreaks with comparable accuracy to traditional methodologies. [3].
One tool that allows users to interact with Internet search data is Google Trends, a free, publically accessible online portal of Google Inc. Google Trends analyzes a portion of the three billion daily Google Search searches and provides data on geospatial and temporal patterns in search volumes for user-specified terms. [4] Google Trends has been used in many research publications, but the range of applications and methods employed have not been reviewed. Furthermore, there are no guidance or agreed standards for the appropriate use of this tool. A critical appraisal of the existing literature would increase awareness of its potential uses in health care research and facilitate a better understanding of its strengths and weaknesses as a research tool.
Accordingly, we performed a systematic review of the health care literature using Google Trends. To characterize how researchers are using Google Trends, we classified studies by topic domain and study aim. We conducted a subanalysis of surveillance studies to further detail their methods and approach to validation. We also assessed the reproducibility of methods and created a checklist for investigators to improve the quality of studies using Google Trends. Finally, we address general limitations in using Google Trends for health care research.

Overview of Google Trends
Google Trends provides access to Internet search patterns by analyzing a portion of all web queries on the Google Search website and other affiliated Google sites. [5] A description of the user interface is shown in Figure S1. Users are able to download the output of their searches to conduct further analyses.
The portal determines the proportion of searches for a userspecified term among all searches performed on Google Search. It then provides a relative search volume (RSV), which is the query share of a particular term for a given location and time period, normalized by the highest query share of that term over the timeseries. [6,7] The user can specify the geographic area to study, whether a city, country, or the world; data is available for all countries worldwide. Furthermore, the user can choose a time period to study, ranging from January 2004 to present, divided by months or days. The user is also able to compare the RSV of up to five different search terms or the RSV of a particular search term between geographic areas and between time periods. In addition, the user can choose from 25 specific topic categories to restrict the search, each with multiple sub categories for .300 choices in total, such as ''Health R Mental Health R Depression''.
With respect to search input, multiple terms could be searched in combination with ''+'' signs and terms can be excluded with ''-'' signs. Quotations can be used to specify exact search phrases. [8].

Study Selection
The review was conducted in accordance with PRISMA guidelines. [9] We included all studies that used Google Trends to answer research questions within the domain of health care. After an initial review, we included letters because they contained substantial original content. We also included studies using Google Insights for Search, a similar tool to Google Trends that was merged into Google Trends in 2012 (hereafter we will refer to studies using Google Insights for Search as using Google Trends for ease of reading).
We excluded studies that primarily focused on Google Flu Trends, a separate tool to specifically track seasonal variation in influenza trends. This tool is distinct from Google Trends and is therefore beyond the scope of this review. We also excluded articles that had no substantial use of Google Trends.  The primary aim of the study as derived from the introduction section of each paper. We abstracted the purpose of each study to provide the reader an overview of how researchers are using Google Trends and to facilitate the categorization of the study aims of Google Trends articles at large.
Methods Variables: We only searched within the methods section of each article for their methodology in using Google Trends. When the exact specification for each category was not in the methods, it was marked as not reported. When any of the specifications were not explicit or clearly evident, it was marked unclear. These variables were directly chosen based on the variables available within the Google Trends portal that could be manipulated by a user, namely location, query category, time period, and search input.

Location Searched
The area that was chosen to study.
Time Period Searched The time period that was chosen to study; this was abstracted in as much detail (month, day) as provided in the article.

Query Category
The query category that was chosen. While the default category is ''all categories'' when using the tool, if this was not explicitly stated, this was marked as not reported.
Google Data Source The use of Google Trends or Google Insights for Search. Google Insights for Search was merged into the Google Trends portal in 2012 and had similar capabilities as Google Trends.

Date of Access
The date that Google Trends or Google Insights for Search was accessed for use. This was looked for either in the methods or in the references of the paper. We abstracted this information because the Google Trends interface and its capabilities have changed over time, and it is important to have a full picture of what a researcher was able to search for at a given time.
Search Input Variables: The search input is comprised of three components: terms utilized, the use of a combination of terms, and the use of quotation marks. Each component is important given how Google Trends works. Each search term input into Google Trends can be a single word or a combination of words (phrase). When a phrase is within quotation marks, the results will only include searches for the exact word order; otherwise, the results include searches of the words in any order. Multiple words in a search term can be separated by a '+' sign, denoting an 'OR' function. On the other hand, a '-' sign separating two words limits the results to searches containing the first word, but excludes searches with the second word. Thus, we chose these three components to abstract.

Terms utilized
The search terms that were input into the search bar to gain output. The exact terms utilized were abstracted, removing any quotation marks provided and other syntax, unless there were more than 15 terms, where only the number of terms was provided. This was a composite, where we abstracted all terms provided in the article and did not differentiate between different searches conducted.

Quotations used
The use of quotations if there are search terms within a search input that are greater than one word. If all the search inputs were single words, this was marked N/A. If there were no quotation marks provided, this was marked as not used. However, if quotation marks were used in the text, but not explicitly and clearly stated that they were used in the search input itself, this was marked unclear. For example, many articles used quotation marks to differentiate terms from the text of the paper but were unclear about use in the search input (e.g. We searched for ''blue'', ''blue dog'', and ''red'').

Combination used
The use of ''+'' or ''-'' marks for search inputs that utilize more than one term. When terms were stated to not be in combination, this was marked as not used. When this was not explicit, this was marked unclear. When there was only one term used for a search input, this field was marked N/ A.

Clear Search Input
We defined clear search input as providing a clear use of quotes or combination when applicable.

Search Rationale
Reasoning provided for any part of the search input (terms used, syntax). If not provided, this was marked as not given. We abstracted this information because the rationale is necessary for a reader to better understand the study methods and to increase the face validity of the study.

Analysis Type
The type of analysis was designated as time trend (using Google Trends data for comparisons across time periods), cross sectional (using Google Trends data for comparisons across different locations at a single time period), or both. We noted all analyses using Google Trends data in the paper. We collected this variable to provide the reader a general perspective on how Google Trends data is being utilized for research.

Primary Findings
The main findings of the paper. If Google Trends-related results were not the primary finding, the findings associated with the tool was also included. We abstracted this information to easily provide the reader a means to understand what each study found.

Number of Citations
The number of citations for an article as determined by Google Scholar on March 9, 2013. We collected this information to assess the leveraging of Google Trends papers by the larger scientific community.
Surveillance Variables: For surveillance studies, we abstracted additional information, as the use of Google Trends data is more nuanced for these studies and the quality and reliability of a surveillance tool depends on methodological rigor and external validation.

Measure/outcome
The phenomena of interest in the study for which surveillance was intended. If multiple, all were noted. When not explicit, these were marked as not given.

Analysis Type
The analysis used for the surveillance portion of the study; not all analyses in the paper.

Time division
The quantum of time (e.g. weekly, monthly) for which Google Trends data was used in time trend analyses. For cross-sectional studies or those that did not use comparison data this was marked N/A.
Geographic division The specific geographic area(s) studied.
Search terms used for surveillance The specific search term(s) assessed for use in surveillance. Where multiple terms were used to produce a single set of predictions, the method by which these were combined is indicated. For example, some studies combined relative search volumes for different terms into a multivariate predictive model. This is distinguished from studies that used a single search volume -based on a combination of multiple search terms -for prediction.
Real-time or Lead-time Whether Google Trends was used to track health phenomena in real time, to predict future patterns, or both. Real-time is using Google Trends data from a given time point/interval to serve as an alert for events occurring at the same time. Lead-time is using Google Trends data from a given time point/interval to predict events occurring at a later time.

Time horizon
The time period over which surveillance was assessed. This may differ from the dates input into Google Trends to extract data (as indicated in Table 2).

Search Strategy
We identified relevant studies by searching Ovid MEDLINE (from inception to January 3, 2014) using a comprehensive search strategy. The list of subheadings (MeSH) and text words used in the search strategy for MEDLINE can be found in Appendix S1. We only included studies of humans written in the English language, and identified 1249 potential articles for inclusion. Since PubMed contains articles from life science journals in addition to articles indexed in MEDLINE, we conducted a search of PubMed (from inception to January 3, 2014) using a similar search strategy, but excluding the articles already identified from MEDLINE. This search identified an additional 871 potential articles, for a total of 2120 potential articles.
Two reviewers (S.V.N. and K.M.) independently reviewed the titles and abstracts of retrieved publications, and 92 articles met our inclusion criteria for full text review. We then excluded 25 studies that did not utilize Google Trends or that met at least one of our exclusion criteria (See Figure 1). We also included 3 articles found from the review of references.
The remaining 70 studies that met our inclusion criteria and did not meet our exclusion criteria form the studies included in this review. Data were abstracted from these studies using a standardized instrument, described below and in Table 1. All extractions were performed by at least two of the authors, and disagreements were resolved by consensus. We did not pool the results due to the heterogeneity of the articles, but we provide summary statistics.

Evaluation of studies
Article Classification. To characterize how researchers are using Google Trends, we created a general descriptive classification of the articles according to their topic domain and study aim using an iterative process. The research team first worked together to examine all of the articles and identify common themes among the articles. We then assigned each article to the themes that emerged. After this initial step, we reassessed these groupings, refining the categorical domains and reassigning articles as needed, to create a classification construct that best characterizes the articles in the review. All disagreements during this process were resolved by consensus. This resulted in four topic domains (infectious disease, mental health and substance use, other noncommunicable diseases, general population behavior) and three study aims (causal inference, description, surveillance). Of note, to categorize study aim, we examined the primary aim of the study as stated by the authors in the introduction of the paper. The definitions of these categories are described in the results.
Variable Abstraction. The variables extracted, along with the standard definitions and rationale for their selection, are listed in Table 1. These pertain to each study's purpose, methodology (search variables, search input, and type of analysis), primary findings, and citations accrued.
We defined whether an article was ''reproducible'' based on whether the authors provided a clear documentation of all fields modifiable by the user, namely location of search, time period of search, query category, and terms utilized, as well as the clear documentation of combination used and quotations used when applicable. Only articles that clearly provided each of these fields (or were deemed not applicable for a given field(s) with all other fields provided) were defined as reproducible. We defined ''clear search input'' as providing a clear use of quotations and/or combination when applicable (see Table 1).
Subanalysis of Surveillance Studies. Following on the popularity of Google Flu Trends, there is particular interest in the potential use of Google Trends data to be operationalized as independent surveillance systems for other diseases. However, such surveillance systems require high standards of testing and validation before being deployed in the real world. Given these particular challenges, we performed a subanalysis of surveillance studies (as determined by study aim), abstracting additional information including the data sources used for validation, the strength of the relationship between predictions and external data, and other methodological characteristics listed in Table 1 and  Table S1. We only assessed if validation data was used in surveillance studies, but we did not assess the quality of the validation process or data.
Assessment of Bias. Conventional tools to assess bias are largely limited to randomized trials and observational studies and are not readily applicable to studies using Google Trends data, which is observational in nature but does not involve individual research participants. [10,11] Therefore, we attempted to address the two primary sources of potential bias within these studies: the search strategy and the validation of surveillance studies. Search methodology may introduce bias, as the selection of terms and changes in search input can alter results. We therefore captured all aspects pertinent to search strategy, including the provision of rationale for search input, for each article. The data sources and methods for validating findings in surveillance studies are also sources for bias, which we assessed in our subanalysis. We assessed for publication bias by evaluating the number of studies with positive findings versus neutral/negative findings.

Study Sample
The 70 articles included in this systematic review are outlined in Table 2. Overall, 92% were original articles; the remaining were letters. Among the articles identified, we observed a seven-fold increase in publications utilizing Google Trends from 2009 to 2013 ( Figure S2). Sixty-three percent of the articles chose to study areas outside of the United States alone. The median number (interquartile range) of article citations was 7 (1,16). The majority of studies (93%) presented positive findings with the tool compared to neutral/negative findings, indicating the possibility of publication bias.

Classification of Published Google Trends Articles
Topic Domain. We classified articles by the primary topic addressed by each article. By consensus we identified four main topic domains: infectious diseases (27% of articles), mental health and substance use (24%), other non-communicable diseases (16%), and general population behavior (33%). The general population behavior category included all health-related behaviors excluding mental health and substance use.
Study aim. There were three categories of study aim: causal inference (27%), description (39%), and surveillance (34%). We defined causal inference studies as those in which the primary aim was to evaluate a hypothesized causal relationship with Google Trends data. An example of a causal inference study is Ayers et al. (2014), who used search queries to explore the potential link between a public figure's cancer diagnosis and population interest in primary cancer prevention. We defined descriptive studies as those that aimed to describe temporal or geographic trends and general relationships, without reference to a hypothesized causal relationship. An example of a descriptive study is Stein et al. (2013), who assessed public interest in LASIK surgery and how levels of interest have changed over time in the United States and other countries. A particular subset of descriptive studies were surveillance studies, which we defined as those in which the stated aim was to evaluate the use of Google Trends to predict or monitor real-world phenomena. An example of a surveillance study is , who assessed whether Google search trends are appropriate for monitoring Norovirus disease.

Methodology of Published Google Trends Articles
Documentation of Search Strategy. Table 3 summarizes the documentation of search strategy. Only 34% of articles documented the date the search was conducted.
Of the variables that can be manipulated within the portal, within the methods section of the papers, 87% documented the location searched, 87% documented the time period searched, and only 19% clearly stated the query category used.
With respect to the search input, only 39% provided a clear search input. Excluding studies with only a single, one-word search term, which are not eligible for using quotations or combinations of terms, only 31% provided a clear search input. Of the articles eligible for using quotations (search terms with .1 word), 81% were unclear and 19% did not use quotations; none provided a clear use of quotations. Of the articles eligible for using a combination of terms (.1 search term used), 31% used a combination, 18% were unclear, and 51% used individual terms.
Reproducibility and Rationale. Overall, only 7% of articles provided requisite documentation for a reproducible search strategy within their methods section; among original articles alone it was 8%.
In addition, we found that only 67% of articles provided a rationale for their search input. Analytic Method. Time trend analysis (comparisons across time periods) was used by 70% of the studies, cross-sectional analysis (comparisons across different locations at a single time period) by 11% of studies, and both by 19%. A variety of analytic methods were used in conjunction with Google Trends output data, including correlation, continuous density hidden Markov models, ANOVA, Box-Jenkins transfer function models, t-tests, autocorrelation, multivariable linear regression, time series analyses, wavelet power spectrum analysis, Cosinor analysis, and the Mann-Whitney test.

Subanalysis of Surveillance Articles and Validation
Among the 24 surveillance studies, 71% used time trend analysis, 25% cross-sectional analysis, and 4% both. Among articles using time trend analysis, 33% utilized lead-time analysis (using Google Trends data from a specific time point/interval to predict events occurring at a later time). Overall, 17% of studies used training/testing data sets and 13% had a time horizon (time period over which surveillance was assessed) ,1 year. More detailed information can be found in Table S1.
With respect to validation, 92% made comparisons against external datasets to validate the Google Trends output; the remaining 8% did not validate their findings. Examples of sources of comparison datasets include disease prevalence data from centers such as the United States Centers for Disease Control, drug revenues from shareholder reports of pharmaceutical companies, and unemployment rates from the Australian Bureau of Statistics.
There was a wide range of correlation statistics, from 0.04 to 0.98 ( Figure S3). Among the 15 papers that used Pearson productmoment correlation, 80% had at least one correlation statistic greater than 0.70.

Checklist for the Documentation of Google Trends Use
In view of the limitations of existing studies identified during this review, we developed a checklist to improve the quality and reproducibility of studies that use Google Trends in the future (Table 4). This was created based directly on the variables that can be manipulated within the Google Trends portal, differences in which could provide differing results among researchers, and the need to provide search strategy rationale. A hypothetical example of a ''well-documented'' search strategy is included within Table 4. Of note, we used brackets to separate the search input from the body text to ensure that the reader understands what was searched for with clear syntax; similar approaches of segregation might be used.

Discussion
In this systematic review of the use of Google Trends in healthcare research, we found that researchers are increasingly utilizing the tool in a diversity of areas in myriad ways; these articles are being widely cited. Furthermore, the majority of surveillance studies validated Google Trends output against external datasets and many had strong correlation statistics. However, the majority of studies lack thorough documentation of search methodologies, which precludes the reproducibility of results; less than 10% of articles are reproducible. In addition, search rationale is often not provided. Thus, while the data within Google Trends holds promise, significant variability and limitations remain around study quality and reliability. The 70 papers included in our review reflected a wide variety of topics and uses. A large proportion of articles used Google Trends to investigate population behavior, which is a logical application of the tool given its basis in user searches. The large proportion of infectious disease articles may stem from the precedent set by Google Flu Trends. [3] Nearly equal numbers of studies used Google Trends for causal inference, surveillance, and description, demonstrating the ability to use the tool to answer a range of questions. There was an increase in publications over time, and the median citation rate (7 per article) is comparable to the average for all scientific articles (7.64 per article). [12] These observations suggest increasing awareness of and the leveraging of information from the tool. Locations studied using the tool were geographically widespread, particularly outside of the United States where conventional data collection may be challenging and resource intensive. Nevertheless, there is evidence of a positive results publication bias, which may be due to the novelty of the tool and authors not submitting and/or editors not accepting negative -and, therefore, perhaps uninteresting -results. [13,14] This publication bias also may be due to researchers retroactively constructing hypotheses about interesting findings after the results are known for a given Google Trends experiment, which can be fast and easy to conduct. [15].
Despite the potential insights and research opportunities that Google Trends provides, many problems were observed with the documentation of methodology. Thorough documentation is necessary to ensure the reproducibility and replicability of the results, which are fundamental tenets of good science. [16] The inability to reproduce studies in the sciences has been welldocumented, and it serves as a central problem to the utility and credibility of research. [17,18] Yet, in our study, only 7% of articles provided clear documentation of the necessary fields to be reproducible. This is especially salient for using Google Trends given the many search fields and multiple options available within each field. Researchers may not have known how to document their methods since this is still a nascent tool for research, without guidance or methodological standards for its use from either Google Inc. or the research community. Furthermore, there were particular issues with the clarity of search inputs. For example, it was often unclear whether quotations provided for a search term were actually used in the search input or were merely given to distinguish the term from the rest of the text. A potential reason for varying presentation of search input syntax may be that possible search syntaxes may have changed over time. [8,19] Nevertheless, the poor documentation of methods also raises larger questions about researchers using Google Trends without knowing the ways in which the tool can be operated.
Different selections of terms to address a common question with Google Trends can produce disparate results and conclusions, and providing the rationale behind these selections is necessary for a reader to better understand the study methods and to increase the face validity of the study. [20] Yet, studies often provided no rationale for their search input. For instance, we do not know why studies chose a given selection of terms or used a specific syntax. Furthermore, there are larger questions about the search strategy as a whole, such as why certain query categories and dates for searching were chosen. Nevertheless, certain studies demonstrated more thorough search strategies and strong rationales for search inputs, particularly accounting for the basis of Google Trends data in user searches. For instance, Desai et al. (2012) included potential misspellings of their search words to fully capture a specific search pattern. In addition, Cho et al. (2013) developed their search inputs by surveying their population of interest, in which they inquired about what search terms subjects would have used to search for influenza. Similar strategies could be adopted by future studies to ensure that their search terms accurately capture the outcome of interest. More guidance is needed by Google to assist researchers in how to produce an optimal search strategy to answer a given question.
Over 90% of surveillance studies compared Google Trends with established data sets, which were often trusted sources of surveillance data. A large number of correlation studies had moderate to strong strengths of association, which demonstrates the potential of Google Trends data to be used for the surveillance of health-related phenomena. For example, Jena et al. (2013) found a strong correlation between searches for HIV and US CDC HIV incidence rates, and were able to construct a model based on searches from years 2007-2008 to accurately predict state HIV incidence for 2009-2010. Moreover, Samaras et al. (2012) showed that Google Trends could have been used to forecast the peak of scarlet fever in the UK 5 weeks before its arrival. Although studies are promising, strong correlations alone do not support the use of Google Trends for surveillance, and further work is needed to substantiate the reliability and real world applicability of Google Trends as a tool to monitor health-related phenomena.
In light of our results, we have proposed a basic checklist for the documentation of Google Trends use. This checklist can serve as a baseline standard to ensure methodological understanding and reproducibility by researchers who choose to use the tool in the future.
While this checklist is a good step forward to improve the reproducibility of results by researchers, there are still larger limitations in the Google Trends tool itself. We cannot clearly ascertain user characteristics and intent from search data, which limits the ability to draw generalizable conclusions about population search behavior. In addition, Google Trends captures the search behavior of only a certain segment of the populationthose with Internet access and using Google Search instead of other search engines. However, the major limitation of Google Trends is the lack of detailed information on the method by which Google generates this search data and the algorithms it employs to analyze it. Furthermore, temporal changes in the interface and capabilities of the Google Trends over time are not documented, which may lead to variation in the search output and therefore study findings.
Moving forward, several steps can be taken to improve the verification of Google Trends study results and the reliability of the tool for research, both on the part of the independent researcher and Google Inc. Researchers should strive to make the raw data they downloaded from the Google Trends available online (as Yang et al., 2010 did [21]) and create an archive or screenshot of the website as they searched it (as Sueki et al., 2011 did [22]) to provide transparency of their methodology and encourage open science with this open tool. Researchers could also evaluate the methods and results of others and themselves to check if there is consistency over time. We encourage Google Inc. to provide a chronology of important changes to Google Trends -in the past and to come -to put researchers' methods and findings in context. Furthermore, if Google Trends continues to be used for research purposes, a discussion and collaboration between Google Inc. and the research community is necessary to create a set of best practices to ensure that the tool is being used responsibly and that its tremendous potential to derive meaningful insights from population search behavior could be fully harvested. While full transparency may not be possible due to commercial sensitivities, informed guidance is needed to ensure the conduct of ethical science. For example, Google Inc. could work together with groups of researchers to detail how to construct optimal queries to fully take advantage of the algorithms at work and to improve the tool to increase the quality of research. In addition, it is important to remember that these conclusions apply not only to Google Trends, but also other similar tools, which currently exist or will likely emerge from existing data sources, that are not intrinsically designed to be utilized for research. In a Big Data era where information and technologies, particularly those that are readily accessible to the public and research community, are growing, mindfulness must be paid to their application in scientific research and efforts must be made to ensure the conduct of good science. One must look no further than the recent controversy around the reliability of Google Flu Trends data to predict influenza incidence and the lack of transparency and inability to verify its results. [23].
Our study has certain limitations. First, given the diversity of topics and uses, there are inherent challenges in the classification of articles. However, at least two independent abstractors reviewed each article and category of abstraction, and disagreements were resolved by group consensus. Second, there are no prior standards to evaluate literature from novel data sources such as Google Trends. Third, our assessment of Google Trends was based on the current syntactic possibilities, but they may have changed over time. [8,19] Conversely, this supports our concerns about undocumented changes to the tool. Finally, there is a possibility that we had an incomplete retrieval of Google Trends articles in our search strategy. However, we conducted an extensive, systematic search of two databases, in addition to reviewing article references, to capture as many articles as possible. Notably, our study focused on the evaluation of the use of Google Trends in research, and we refrained from making any commentary about the conclusions drawn by researchers in these studies. Further studies are needed to rigorously evaluate the interpretations of causal inference studies and the validity of Google Trends for surveillance.

Conclusion
Google Trends holds potential as a free, easily accessible means to access large population search data to derive meaningful insights about population behavior and its link to health and health care. However, to be reliably utilized as a research tool, it would have to be more transparent, which will increase the trustworthiness of both the results generated and its general applicability for health care research. Furthermore, researchers must make efforts to clearly state their rationale and document their experiments to ensure the reproducibility of results. The lessons gleaned from this review are also instructive for other tools not intrinsically designed for research that may emerge in an era of Big Data to ensure that they are used appropriately by the scientific community.

Supporting Information
Appendix S1 MEDLINE Search Strategy.