Individuals with TIA/stroke symptoms often do not seek urgent medical attention. We assessed the feasibility of identifying individuals searching for information on TIA/stroke symptoms online as a target for future interventions to encourage urgent evaluation and we evaluated the performance of a self-reported risk score to identify subjects with true TIA or stroke.
We placed online advertisements to target English-speaking adults in the United States searching for TIA/stroke-related keywords. After completing an online questionnaire, participants were telephoned by a vascular neurologist to assess the likelihood of TIA/stroke. We used logistic regression and the c-statistic to assess associations and model discrimination respectively. Over 122 days, 251 (1%) of 25,292 website visitors completed the online questionnaire and 175 were reached by telephone (mean age 58.5 years; 63% women) for follow-up. Of these participants, 37 (21%) had symptoms within 24 hours, 60 (34%) had not had a medical evaluation yet, and 68 (39%) had TIA/stroke. Applying a modified ABCD2 score yielded a c-statistic of 0.66, but 2 of 12 with a zero score had a TIA/stroke. Those with new symptoms were more likely to have TIA/stroke (OR 4.90, 95% CI 2.56−9.09).
Individuals with TIA/stroke that are seeking real-time information on symptoms online can be readily identified, in some cases before they have sought formal medical evaluation. Although a simple self-reported risk score was unable to identify a low-risk population in this selected group, this population may still present an attractive target for future interventions designed to encourage urgent medical evaluation.
Citation: Kim AS, Poisson SN, Easton JD, Johnston SC (2012) A Cross-Sectional Study of Individuals Seeking Information on Transient Ischemic Attack and Stroke Symptoms Online: A Target for Intervention? PLoS ONE 7(10): e47997. https://doi.org/10.1371/journal.pone.0047997
Editor: Jens Minnerup, University of Münster, Germany
Received: April 27, 2012; Accepted: September 25, 2012; Published: October 31, 2012
Copyright: © 2012 Kim et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This study was sponsored by the National Stroke Association through a grant from Boehringer Ingelheim Pharmaceuticals, Inc. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Transient ischemic attack (TIA) is associated with a high risk of subsequent stroke, particularly in the first hours after symptom onset. ,  Although progress has been made to develop simple clinical scores to rapidly identify TIA patients at high risk of stroke  and to apply urgent interventions to prevent stroke after TIA, , ,  most individuals with TIA do not seek urgent medical evaluation. , .
Knowledge of TIA among the general public is poor.  TIA or “mini-stroke”, as it is commonly known in the lay public, may not be recognized as a harbinger of stroke, particularly when symptoms are transitory or evanescent. ,  So even when TIA or stroke is considered as a possibility, the importance of an urgent evaluation may not be fully appreciated. Furthermore, individuals seeking appropriate evaluation may be deterred by barriers to access to healthcare providers, particularly on weekends or evenings  or by the prospect of lengthy or costly emergency department visits for symptoms that may be perceived of as minor or transient. ,  Therefore, improving early recognition of TIA and an appreciation for the substantial risk of stroke associated with TIA may help to reduce delays in presentation for medical evaluation and create additional opportunities to prevent stroke.
Within this context, the internet has become an increasingly important source of first-line medical information  in part due to widespread availability and relatively low barriers to access.  However, despite potential advantages in terms of cost, efficiency, and reach, internet-based public health interventions to target online information-seeking behavior have been underdeveloped and understudied.
Our ultimate goal is to reduce the burden of disease from stroke by developing and refining efficient interventions to empower individuals seeking information on TIA/stroke online to seek urgent medical attention. For the present study, our primary goal was to assess the feasibility of identifying a population of individuals that are searching for information on TIA/stroke online as a potential target for future interventions to encourage appropriate urgent medical evaluation. Our main target population was subjects with TIA since these patients are at higher short-term risk of developing a stroke than patients with prevalent stroke, but since the distinction between TIA and stroke is difficult to make by both clinicians and the lay-public, we also included patients with prevalent stroke. Our secondary goal was to assess the performance of a web-based self-assessment tool based on the ABCD2 score to identify TIA/stroke  in this population with the hope that this personalized information would be a more effective motivation for participants to seek appropriate medical attention than general advice on TIA.
We conducted a cross-sectional study with recruitment, eligibility screening, enrollment, consent, and data collection activities accomplished online and with outcome assessments completed by telephone.
Instrument Development and Pre-Testing
Candidate items for the online instrument were drawn from the Questionnaire for Verifying Stroke-Free Status;  components of the ABCD2 score (Age > = 60 = 1; Clinical Features: Speech Disturbance without weakness = 1, Unilateral Weakness = 2; Duration: 10–59 minutes = 1, >60 minutes = 2; Diabetes = 1);  and other items (frequency and timing of symptoms, suddenness of symptom onset, hypertension, and migraine history). The blood pressure item was omitted because this information would usually not be available by self-report for most participants. We pre-tested candidate items in 10 inpatients and 10 outpatients with confirmed TIA or minor stroke using semi-structured cognitive interviews to assess for comprehension and phrasing. This feedback was incorporated into the final instrument (see Text S1).
Recruitment, Screening, Consent, and Enrollment
We placed a series of advertisements using an online marketing tool (Google Adwords, Google, Mountain View, California) to target users in the United States that were searching for keywords related to “mini-stroke”, “TIA”, “transient ischemic attack” or that were visiting webpages with related content that were part of an online advertising network (Google Display Network; See Text S2 for a list of targeted keywords). Users that clicked on an advertisement were taken to the study homepage (http://tia.ucsf.edu/), which provided information about the research study and basic information about stroke and TIA. Users that were interested in participating in the study then completed a seven-item screening questionnaire to confirm eligibility (symptoms concerning for a transient ischemic attack or “mini-stroke” within the last 6 months; age > = 18; ability to read, speak, and understand English; residence in the United States; and a willingness to complete the study questionnaire and the follow-up telephone assessment.) Those who met all entry criteria were asked to provide informed consent online. Consent was confirmed using a five-item post-test of comprehension. Those who were unwilling or unable to provide consent were excluded.
Participants provided contact information for the subsequent telephone assessment and then completed a 26-item questionnaire on demographic characteristics, medical history, and symptoms (See Text S3). We then scheduled a 15 minute telephone call with each participant in order to assess how likely the symptoms were due to TIA/stroke. These assessments were conducted by one of five vascular neurologists. Each neurologist was masked to the previously submitted responses to the questionnaire. Neurologists were free to elicit any information in order to assess the likelihood of TIA/stroke including self-reported neuroimaging findings. The likelihood of TIA/stroke was rated as “definitely not”, “unlikely”, “probable”, and “definite”.
TIA was defined as an acute neurological deficit lasting less than 24 hours attributable to focal brain ischemia, without evidence of a non-ischemic etiology and stroke was defined as a new neurologic deficit lasting over 24 hours with no apparent clinical or radiologic indication of a nonvascular mimic. To assess interrater reliability, half of the participants had a second independent telephone evaluation by a separate vascular neurologist that was masked to the results of the first assessment.
Based on questionnaire responses, we calculated a modified ABCD2 score for each participant during the analysis phase of the study. We used logistic regression to assess univariate and multivariable associations. Cronbach’s alpha was used to assess reliability across items assessing the same underlying construct (i.e. diabetes, hypertension, suddenness of symptom onset). We used a quadratic-weighted kappa to evaluate interrater agreement between the two neurologists.
Recruitment and Enrollment
A total of 4,600,427 advertisements were displayed over the 122-day enrollment period which resulted in 26,602 visits (average 218 visits/day) from 25,292 unique users. Total advertising costs for the study were $8,798.74–which corresponds to $0.35 per unique visitor or $35.05 per enrollee. All fifty states were represented as assessed by an Internet Protocol address-based geolocalization algorithm.
Fully 86% percent of visitors immediately exited the website within a few seconds of arriving at the home page, which is consistent with the notion that they clearly did not qualify for the study or were not interested in enrolling. Of the 291 visitors that initiated the enrollment process, 251 (86%; 1% overall) completed the eligibility screening, online consent, and questionnaire, which is not an unexpected yield rate for internet advertisements generally since there is trivial effort and little commitment for users to click on a link to find out more information. Of these, 175 (70%) were reached by telephone for follow-up. Reasons for failing to contact participants by telephone included the following: 31 did not respond to multiple telephone calls; 14 were initially reached by telephone to confirm their participation, but did not respond to subsequent telephone calls; 6 withdrew consent when reached; and 25 listed a nonworking or incorrect telephone number.
The average age of participants was 59 and a majority were women (Table 1). A total of 52 participants (30%) were age 65 or older. Based on telephone area codes, participants from 40 states and the District of Columbia were represented. Vascular risk factors such as hypertension, diabetes, and a prior history of cerebrovascular disease were common. A prior history of migraine was also frequently reported.
Speech, motor, and sensory symptoms were commonly reported (Table 1). Double vision and associated headache were also commonly reported as well. A total of 102 (58%) participants reported that symptoms were sudden in onset (onset over seconds to minutes) and 31 (19%) reported having a warning or prodrome of impending symptoms. Some participants (71; 41%) reported symptoms that had lasted for more than an hour.
A majority of participants reported that the symptoms were new within the past year (104; 59%) but 26 (15%) reported that they had had more than 11 episodes in the past year. Most participants (131; 77%) had experienced symptoms within a month and 37 (22%) had experienced symptoms within 24 hours. A substantial number of participants (71; 41%) had not yet sought formal medical advice.
Based on the follow-up telephone calls, 43 (25%) participants had a probable or definite TIA, and 68 (39%) participants had a probable or definite cerebrovascular event (either stroke or TIA). There was 96% agreement between the two vascular neurologists for the diagnosis of TIA/stroke (quadratic-weighted kappa = 0.86; see Table S1) among the 84 (48%) participants that had a second independent assessment.
Assessment of ABCD2 Score Components
Univariable associations between individual components of the ABCD2 score and TIA/stroke are presented in Table 2. A longer duration of symptoms was significantly associated with the ultimate diagnosis of TIA/stroke (Table 2). Unilateral weakness was also associated with a nearly 2-fold higher odds of TIA/stroke (OR 1.95, 95% CI 1.08−3.68).
The mean modified ABCD2 score (excluding blood pressure) was 2.7, (SD 1.4; range 0–6). The proportion of patients with a TIA or stroke stratified by risk score is shown in Figure 1. Of the 12 patients with a score of 0, two (17%) had a TIA/stroke–one participant had diplopia and unilateral facial numbness that was concerning for posterior circulation ischemia, and the other had amaurosis fugax.
A modified ABCD2 score was calculated from self-reported components of the score (age > = 60 = 1 point; Clinical Features: Speech Disturbance without weakness = 1 point, Unilateral Weakness = 2 points; Duration: 10–59 minutes = 1 point, >60 minutes = 2 points; Diabetes = 1 point). The blood pressure item was excluded since this was not likely to be available by self-report, so the range of this risk score was 0 to 6. The grey bars show the proportion observed and the black vertical lines represent the 95% confidence interval around that proportion. The one enrollee with a score of 6 had migraine.
Participants with multiple episodes of similar symptoms within the past year were much less likely to be diagnosed with TIA/stroke (Table 2). The odds of a TIA/stroke diagnosis were nearly five times higher (OR 4.90, 95% CI 2.56−9.09) for participants with new symptoms within the last year as compared to those who had at least one previous episode of similar symptoms in the last year.
There was moderate agreement (Cronbach’s alpha 0.79) among the three items on the sudden symptom onset. However, sudden symptom onset was not significantly associated with a TIA/stroke diagnosis. Those who had already sought medical advice for their symptoms were significantly more likely to have TIA/stroke (OR 2.77, 95% CI 1.38–5.59).
For Model 1, we included components of the ABCD2 score except of blood pressure (Table 3). This self-reported risk score demonstrated poor discrimination for TIA/stroke (0.66, 95% CI 0.58–0.74).
For Model 2, we incorporated a categorical variable with the number of episodes of similar symptoms within the past year (Table 3). This model produced a higher c-statistic (0.76, 95% CI 0.68–0.83) for TIA/stroke. However the same two participants with a risk score of 0 described above would still have been classified in the lowest risk category.
Using an efficient and largely automated approach, we were able to readily identify a selected group of individuals who were seeking information on recent symptoms of TIA/stroke online–sometimes very soon after symptom onset and often before a formal medical evaluation had been completed. Although participants in our study are unlikely to be fully representative of patients with TIA generally, or patients seeking online information on TIA specifically, we did identify a subpopulation with a relatively high burden of cerebrovascular disease that could benefit from a targeted public health intervention. Given the low cost and the substantial efficiency of this approach in terms of cost per participant reached, and since a major goal of acute management of TIA is urgent evaluation to prevent early recurrent stroke, this subpopulation may present an attractive target for interventions designed to reduce delays in presenting to medical attention for TIA/stroke.
A secondary goal of our study was to assess the performance of a self-reported version of ABCD2 score for TIA. The ABCD2 score was originally developed to assess the risk of stroke after TIA using simple elements available to front-line clinicians,  but this score has also been shown to work in part by distinguishing TIA from TIA mimics.  Here, we faced the additional challenge from the measurement issues that arise from using self-reported responses from individuals with various levels of health literacy. Accordingly, we found that a modified ABCD2 score by self-report could not reliably rule out a TIA or stroke. Some participants (2 of 12) with the lowest score of 0 had symptoms such as diplopia or amaurosis fugax that are not specifically captured by the score, but would still require urgent evaluation.
With regards to the performance of other self-reported items to predict a cerebrovascular diagnosis, we found a strong association between a higher number of episodes of similar symptoms within the last 12 months and a non-cerebrovascular diagnosis. Having multiple stereotyped episodes of symptoms previously without a resultant stroke may make a given episode less likely to be TIA and may make seizure, migraine or other episodic disorders more likely.  The predictive value of this finding will require further validation and confirmation of the dose-response relationship that is suggested by our data.
Since such a high proportion of participants in our study had TIA or stroke, we speculate that all of the individuals that participated in our study may have been well served to seek medical attention regardless of risk score, particularly since the online questionnaire was unable to reliably identify low-risk individuals. Some individual participants indicated that they lacked a primary care provider or medical insurance or otherwise had limited access to medical care, while others indicated that they did not understand that TIA was a medical emergency, or were unable to get an urgent appointment, but had already planned on seeking medical attention at some point. The value of some form of risk stratification at the outset may not be to identify those without worrisome symptoms, but to provide some interactivity and engagement in order to enhance behavior change while participants are deciding on next actions. We suspect that if individuals are given feedback and recommendations that are to some extent individualized and interactive rather than presented as static blanket recommendations, they may be more likely to take action after TIA, though a formal assessment of this possibility would require additional study.
Our methods also highlight opportunities to improve the efficiency of clinical research more generally by quickly and efficiently reaching hundreds of potential participants on a daily basis and by enrolling participants over a large geographic area. Direct-to-participant recruitment portals that incorporate initial eligibility screening by self-report may demonstrate a sufficiently high yield for TIA and stroke diagnoses to serve as a primary recruitment method for some types of clinical research. Furthermore, our study frames the potential role for validated and interactive self-assessment online tools address public health problems more generally. A previous study have demonstrated a relationship between internet search queries and influenza epidemics  and another study showed that the frequency of internet search queries for stroke related search terms by state (“stroke signs”, “stroke symptoms”, “mini stroke”, excluding “heat”) was associated with stroke prevalence.  There remain potential medicolegal issues including whether providing tailored or personalized information constitutes medical advice, and valid concerns about maintaining privacy and confidentiality of health information online. But as disparities in access , ,  and adoption of internet technology across demographic and sociodemographic strata narrow,  internet-based interventions to efficiently target public health problems may become more attractive. Future studies may include questionnaires in the ED regarding internet-based activity prior to presentation in the ED and evaluations of the impact of internet-based tools on the actual the behavior of subjects with possible TIA and ultimately on processes measures of treatment and patient outcomes.
Our study should be interpreted in light of a number of limitations. First, about 1% of visitors to the website enrolled in the study and so the participants in this study were highly selected. Although the motivations for enrollment were not captured in our study, many visitors quickly moved from the website within a few seconds, which suggests that they were not finding what they had expected or what they were looking for. We speculate that individuals with symptoms without a straightforward diagnosis despite prior medical evaluations, those with chronic medical conditions, or those with barriers to accessing care might be overrepresented in our study. Second, the authentication of participants and their responses and the difficulty of reaching some participants by telephone limit the generalizability of our results and illustrate a particular challenge for internet-based research. Third, we attempted to mitigate misclassification of TIA and stroke outcomes by formally assessing interrater reliability and masking the neurologists to questionnaire responses, but since follow-up was conducted by telephone and access to additional clinical information such as neuroimaging studies or results of a neurologic exam was limited to self-report the potential for misclassification remains. For the lay-public, the use of the term, “mini-stroke,” a much more common search term than “TIA”, serves to confuse the distinction between TIA and stroke and the clinical distinction between TIA and minor stroke may be difficult to make even for experienced clinicians. ,  Fourth, although the neurologists were masked to responses to the questionnaire, they were free to elicit any information into their final assessments and may have incorporated items that were elicited by the online questionnaire into their assessments.
Individuals seeking information on possible TIA or stroke symptoms online can be efficiently and readily identified, sometimes before they have sought formal medical attention. Although elements of a self-reported ABCD2 score was not able to rule out a cerebrovascular etiology, the burden of true stroke and TIA in this selected subpopulation may be sufficiently high to justify future targeted interventions to encourage urgent medical evaluation.
Interrater agreement on cerebrovascular diagnosis. Agreement between two independent telephone assessments by vascular neurologists for the diagnosis of stroke/TIA among 84 participant seeking information on TIA/stroke symptoms on the internet. Agreement for Stroke/TIA was 95.8% and the quadratic-weighted kappa was 0.86.
Target Search Terms for Targeting Internet Advertisements.
We thank Christine Wong, MD (California Pacific Medical Center, data collection) and Babak Navi, MD (Weill Cornell Medical College, data collection) for assistance conducting follow-up telephone assessments.
Conceived and designed the experiments: ASK SCJ. Performed the experiments: ASK SNP JDE SCJ. Analyzed the data: ASK. Wrote the paper: ASK SNP JDE SCJ.
- 1. Johnston SC, Gress DR, Browner WS, Sidney S (2000) Short-term prognosis after emergency department diagnosis of TIA. JAMA 284: 2901–2906.
- 2. Coull AJ, Lovett JK, Rothwell PM (2004) Population based study of early risk of stroke after transient ischaemic attack or minor stroke: implications for public education and organisation of services. BMJ 328: 326.
- 3. Johnston SC, Rothwell PM, Nguyen-Huynh MN, Giles MF, Elkins JS, et al. (2007) Validation and refinement of scores to predict very early stroke risk after transient ischaemic attack. Lancet 369: 283–292.
- 4. Lavallée PC, Meseguer E, Abboud H, Cabrejo L, Olivot J-M, et al. (2007) A transient ischaemic attack clinic with round-the-clock access (SOS-TIA): feasibility and effects. Lancet Neurol 6: 953–960.
- 5. Rothwell PM, Giles MF, Chandratheva A, Marquardt L, Geraghty O, et al. (2007) Effect of urgent treatment of transient ischaemic attack and minor stroke on early recurrent stroke (EXPRESS study): a prospective population-based sequential comparison. Lancet 370: 1432–1442.
- 6. Giles MF, Rothwell PM (2007) Risk of stroke early after transient ischaemic attack: a systematic review and meta-analysis. Lancet Neurol 6: 1063–1072.
- 7. Chandratheva A, Lasserson DS, Geraghty OC, Rothwell PM (2010) Population-based study of behavior immediately after transient ischemic attack and minor stroke in 1000 consecutive patients: lessons for public education. Stroke 41: 1108–1114.
- 8. Sprigg N, Machili C, Otter ME, Wilson A, Robinson TG (2009) A systematic review of delays in seeking medical attention after transient ischaemic attack. J Neurol Neurosurg Psychiatry 80: 871–875.
- 9. Johnston SC, Fayad PB, Gorelick PB, Hanley DF, Shwayder P, et al. (2003) Prevalence and knowledge of transient ischemic attack among US adults. Neurology 60: 1429–1434.
- 10. Mandelzweig L, Goldbourt U, Boyko V, Tanne D (2006) Perceptual, social, and behavioral factors associated with delays in seeking medical care in patients with symptoms of acute stroke. Stroke 37: 1248–1253.
- 11. Lasserson DS, Chandratheva A, Giles MF, Mant D, Rothwell PM (2008) Influence of general practice opening hours on delay in seeking medical attention after transient ischaemic attack (TIA) and minor stroke: prospective population based study. BMJ 337: a1569.
- 12. Maestroni A, Mandelli C, Manganaro D, Zecca B, Rossi P, et al. (2008) Factors influencing delay in presentation for acute stroke in an emergency department in Milan, Italy. Emerg Med J 25: 340–345.
- 13. Giles MF, Flossman E, Rothwell PM (2006) Patient behavior immediately after transient ischemic attack according to clinical characteristics, perception of the event, and predicted risk of stroke. Stroke 37: 1254–1260.
- 14. Lustria ML, Smith SA, Hinnant CC (2011) Exploring digital divides: an examination of eHealth technology use in health information seeking, communication and personal health information management in the USA. Health informatics journal 17: 224–243.
- 15. Hesse BW, Nelson DE, Kreps GL, Croyle RT, Arora NK, et al. (2005) Trust and sources of health information: the impact of the Internet and its implications for health care providers: findings from the first Health Information National Trends Survey. Archives of internal medicine 165: 2618–2624.
- 16. Palomeras E, Fossas P, Quintana M, Monteis R, Sebastian M, et al. (2008) Emergency perception and other variables associated with extra-hospital delay in stroke patients in the Maresme region (Spain). Eur J Neurol 15: 329–335.
- 17. Jones WJ, Williams LS, Meschia JF (2001) Validating the Questionnaire for Verifying Stroke-Free Status (QVSFS) by neurological history and examination. Stroke 32: 2232–2236.
- 18. Josephson SA, Sidney S, Pham TN, Bernstein AL, Johnston SC (2008) Higher ABCD2 Score Predicts Patients Most Likely to Have True Transient Ischemic Attack. Stroke 39: 3096–3098.
- 19. Johnston SC, Sidney S, Bernstein AL, Gress DR (2003) A comparison of risk factors for recurrent TIA and stroke in patients diagnosed with TIA. Neurology 60: 280–285.
- 20. Ginsberg J, Mohebbi MH, Patel RS, Brammer L, Smolinski MS, et al. (2009) Detecting influenza epidemics using search engine query data. Nature 457: 1012–1014.
- 21. Walcott BP, Nahed BV, Kahle KT, Redjal N, Coumans JV (2011) Determination of geographic variance in stroke prevalence using Internet search engine analytics. Neurosurgical focus 30: E19.
- 22. Choi N (2011) Relationship between health service use and health information technology use among older adults: analysis of the US National Health Interview Survey. Journal of medical Internet research 13: e33.
- 23. Wasserman IM, Richmond-Abbott M (2005) Gender and the Internet: Causes of Variation in Access, Level, and Scope of Use*. Social Science Quarterly 86: 252–270.
- 24. Zach L, Dalrymple PW, Rogers ML, Williver-Farr H (2012) Assessing internet access and use in a medically underserved population: implications for providing enhanced health information services. Health information and libraries journal 29: 61–71.
- 25. Ferro JM, Falcao I, Rodrigues G, Canhao P, Melo TP, et al. (1996) Diagnosis of transient ischemic attack by the nonneurologist. A validation study. Stroke 27: 2225–2229.
- 26. Quinn TJ, Cameron AC, Dawson J, Lees KR, Walters MR (2009) ABCD2 scores and prediction of noncerebrovascular diagnoses in an outpatient population: a case-control study. Stroke 40: 749–753.