During an influenza pandemic, the United States Centers for Disease Control and Prevention (CDC) may recommend school closures. These closures could have unintended consequences for students and their families. Publicly available social media could be analyzed to identify the consequences of an unplanned school closure.
As a proxy for an unplanned, pandemic-related school closure, we used the district-wide school closure due to the September 10–18, 2012 teachers’ strike in Chicago, Illinois. We captured social media posts about the school closure using the Radian6 social media-monitoring platform. An online workforce from Amazon Mechanical Turk categorized each post into one of two groups. The first group included relevant posts that described the impact of the closure on students and their families. The second group included irrelevant posts that described the political aspects of the strike or topics unrelated to the school closure. All relevant posts were further categorized as expressing a positive, negative, or neutral sentiment. We analyzed patterns of relevant posts and sentiment over time and compared our findings to household surveys conducted after other unplanned school closures.
We captured 4,546 social media posts about the district-wide school closure using our search criteria. Of these, 930 (20%) were categorized as relevant by the online workforce. Of the relevant posts, 619 (67%) expressed a negative sentiment, 51 (5%) expressed a positive sentiment, and 260 (28%) were neutral. The number of relevant posts, and especially those with a negative sentiment, peaked on day 1 of the strike. Negative sentiment expressed concerns about childcare, missed school lunches, and the lack of class time for students. This was consistent with findings from previously conducted household surveys.
Social media are publicly available and can readily provide information on the impact of an unplanned school closure on students and their families. Using social media to assess the impact of an unplanned school closure due to a public health event would be informative. An online workforce can effectively assist with the review process.
Citation: Rainey JJ, Kenney J, Wilburn B, Putman A, Zheteyeva Y, O’Sullivan M (2016) Online Work Force Analyzes Social Media to Identify Consequences of an Unplanned School Closure – Using Technology to Prepare for the Next Pandemic. PLoS ONE 11(9): e0163207. https://doi.org/10.1371/journal.pone.0163207
Editor: David L. Swerdlow, Pfizer Inc, UNITED STATES
Received: August 19, 2015; Accepted: September 6, 2016; Published: September 21, 2016
This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Closing schools can help slow influenza transmission among a school-aged population and is an important mitigation measure during the early stages of an influenza pandemic [1–4]. However, unplanned school closures can also cause economic and social costs and consequences for students and their families , especially if the closure lasts for several weeks or more. Public health officials must therefore carefully evaluate the balance between the benefits and the consequences of these closures to mitigate the spread pandemic influenza [1, 5]. Being aware of these costs and consequences during the course of a pandemic could assist public officials to better evaluate this balance.
Since influenza pandemics are infrequent in the United States, public health officials have relied on investigating the costs and consequences of unplanned school closures due to non-pandemic related causes. Although many of these investigations have been informative, some were implemented several months after schools reopened [6–8], possibly resulting in recall bias. Others were conducted during infectious disease outbreaks or immediately after, but the investigations still required time and resources for planning, implementation, and analysis before results became available [9–11]. During the 2009 Influenza A (H1N1) pandemic, telephone polls were conducted to identify challenges to families associated with recent school closures. These polls were substantially more time-efficient but only captured information for a single point in time . All types of household surveys and telephone polls create some level of burden on household respondents.
The accessibility and popularity of social media such as Twitter, Facebook, and blogs provide a new opportunity to assess public perception and the impact of community-wide events [13–21]. In a 2014 study, 74% of online adults used social media , suggesting that a high volume of potentially relevant information is available in the public domain. These sources could capture nearly real-time information (in relation to the community event) in unconstrained formats while minimizing common biases (e.g., recall bias) and the burden associated with traditional survey methods [13–18, 23–24]. During a pandemic, monitoring public perception and sentiment over time could help determine when modifications or different strategies may be needed for mitigating disease transmission.
We analyzed social media related to the district-wide unplanned school closure due to the September 2012 Chicago teacher’s strike, which affected more than 400,000 students attending the 600 elementary, middle, and high schools in the district . Our objectives included: 1) evaluating whether social media could be used to identify costs and consequences as effectively as traditional household surveys and telephone polls; 2) determining if the costs and consequences identified varied over the duration of an unplanned school closure; and 3) exploring the use of an online workforce as a way to efficiently review and interpret relevant social media posts. This type of workforce could be easily leveraged if needed during a future pandemic or other public health emergency.
We used the Radian6 (San Francisco, CA) social media-monitoring platform to retrospectively capture social media posts about the Chicago City School District closure occurring from September 10–18, 2016. Since the strike was anticipated prior to the first day of the closure, we captured social media posts from Twitter, Facebook, blogs, forums, and comments between September 8 and September 21, (two days before the strike started to three days after the strike ended). We used the following combination of exact search terms: “strike Chicago” AND “breakfast” OR “childcare” OR “daycare” OR “lunch” OR “parent”. A proximity score of “5” was applied to the terms “strike” and “Chicago” (on a scale of 1–20, with 1 being exact [i.e., strike and Chicago together]). We included childcare and missed free or reduced-priced school lunches since these issues were identified in previous unplanned school closure investigations. Several different combinations of search terms were tested to capture relevant information while limiting unnecessary noise (e.g., posts related to political aspects of the strike). With the above Boolean logic, only “strike Chicago” and one of the other terms were required in any of the identified posts.
Posts with poor agreement among the Turk workers (< 80%) were subsequently reviewed by four subject matter experts (SMEs) comprising public health staff with training in reviewing social media posts and investigating unplanned school closures. The final categorization of these posts was determined by a majority of these experts. In the SME review, we included a 5% random sample of posts (n = 180) previously categorized as irrelevant by the Turk workers to assess the agreement between the SMEs and recruited Turk workers. All posts categorized as relevant by Turk workers were also reviewed and verified by the SMEs.
Using a combination of context (e.g., school lunches) and semantics (e.g., wonderful/terrible), each relevant post was categorized into one of three sentiment groups [16, 19]: positive, negative, or neutral. These groups were defined as follows:
- Positive: The author expressed a good or favorable experience as a result of the closure. Example of positive post: “Another day without school, a day to play.”
- Negative: The author expressed inconveniences or an undesirable effect as a result of the closure. Example of negative post: “I can’t find childcare.”
- Neutral: The author did not express any particular sentiment. Example of neutral post: “Schools will be open at 8:00 to serve breakfast to students.”
We tested and modified project definitions before initiating the sentiment analysis to maximize clarity and understanding. Posts stating “school will be closed today” or describing the availability of services were considered statements of fact and were categorized as neutral. When both positive and negative sentiments were expressed, the post was categorized as negative since our overall objective was to describe the costs and consequences of an unplanned school closure.
We described the temporal trends of relevant posts and their sentiments to assess whether perceptions and sentiments changed during the 10-day closure. We also abstracted up to three consequences from each post categorized as having a negative sentiment. We qualitatively compared these findings with results from recent household surveys and a telephone poll following other unplanned school closures. The similarities and differences were used to assess the validity of our approach. Additionally, we calculated overall and daily sentiment scores as the difference between the number of positive and negative sentiment posts divided by the sum of all relevant posts ((positive—negative)/(positive + negative + neutral)) . A score less than zero suggested negative sentiment, while a score greater than zero suggested positive sentiment . Data were analyzed using SAS (version 9.3, Cary, NC).
We retrospectively captured 4,546 social media posts for the dates of September 8, 2012 to September 21, 2012 using our search criteria. Of these, 930 (20%) were categorized as relevant in describing the impact of the strike-related closure on students and their families (Table 1). The remaining 3,616 (80%) social media posts were excluded as irrelevant. The 4,546 posts were reviewed by 301 different Turk workers (each categorizing 1 to 1,532 posts such that every post was reviewed by five different workers). The median number of posts categorized per worker was 17 (IQR: 3 to 71), and the median time for categorizing each post was 1.7 minutes (IQR: 13 seconds to 5 minutes). On average, Turk workers completed the review of each batch of 200 posts within 2–3 hours. The agreement between Turk workers and SMEs was 99% for irrelevant posts (two of the subset of 180 posts were re-categorized as relevant by experts) and 98% for relevant posts (16 of 944 posts were re-categorized as irrelevant). Almost 90% of the 930 relevant posts were from Twitter, blogs and comments, Facebook discussions, and mainstream media comments (Table 1).
Relevant posts further categorized according negative, positive, and neutral sentiment. Posts captured from social media referencing Chicago teachers’ strike from September 8–21, 2012 (two days before to three days after strike).
Of the relevant posts, 619 (67%) were further categorized as expressing negative sentiment, 51 (5%) were positive, and 260 (28%) were neutral. We abstracted 1,007 costs and consequences from the 619 negative posts. The most frequently expressed negative sentiment involved concerns about finding childcare or the cost of childcare (n = 377), missing school or class time (n = 210), and missing free or reduced-priced school meals (n = 185) (Table 2). One parent posted, “both of us are working, I have no relatives, nobody I can turn to in town,” while another expressed, “besides the daycare issue, they just need to be in school.” These costs and consequences were consistent with findings from other unplanned school closure investigations (Table 3). Additional consequences were captured for the unplanned school closure in Chicago, including concerns with child safety and city violence (n = 45). Social media provided information for parents on locating childcare and alternative meal services for students (e.g., “CTU has announced their strike. Visit http://t.co/TVptuuTm or call 311 if you have no alternative childcare tomorrow.”).
Each post could express up to three costs and consequences. Posts captured from social media referencing Chicago teachers’ strike from September 8–21, 2012 (two days before to three days after strike).
The number of all relevant posts and those expressing negative sentiment was greatest on day 1 of the strike (September 10, 2012) and decreased dramatically by day 3 (September 13, 2012 [Fig 1]). The sentiment score was estimated to assess the relative change in negative versus positive sentiment over time. During the Chicago teachers’ strike, the mean sentiment score was -0.61 (daily range: -0.20 to -1.0 [Fig 2]).
Relevant posts mentioned impact of unplanned school closure due to Chicago teachers’ strike from September 8–21, 2012 (two days before and three days after strike) on students and their families (n = 930). aSentiment definitions: Positive: The author expressed a good or favorable experience as a result of the closure. Example of positive post: “Another day without school, a day to play.” Negative: The author expressed inconveniences or undesirable effect as a result of the closure. Example of negative post: “I can’t find childcare”. Neutral: The author did not express any particular sentiment. Example of neutral post: “Schools will be open at 8:00 to serve breakfast to students”.
Relevant posts mentioned impact of unplanned school closure due to Chicago teachers’ strike from September 8–21 (two days before to three days after strike) on students and their families (N = 930). Sentiment score was calculated as: (positive posts—negative posts)/(positive posts + negative posts + neutral posts)a-b. aSentiment score < 0 suggests negative sentiment; score > 0 suggests positive sentiment. bScore on September 21 reflects only four relevant posts, all expressing negative sentiment.
Reviewing and analyzing social media posts at the time of the Chicago teachers’ strike generated information on the consequences and community sentiment of this unplanned district-wide school closure. The types of identified consequences were similar to findings from previous unplanned school closure investigations and included missing work, childcare costs, and lack of access to free and reduced-priced school lunches [6–9]. We also captured social media voicing concerns about child safety and street violence, neither of which had been previously identified as possible consequences of unplanned school closures. These newly identified concerns and consequences may highlight the importance of capturing local information when assessing the impact of unplanned school closures on students and their families. The unstructured format of social media posts can elicit information not previously anticipated [13–17, 23], and is likely an added benefit of social media when compared to traditional surveys and telephone polls.
We were also able to capture information on the costs and consequences as well as sentiment in near real time throughout the strike. Relevant posts and negative posts were most frequent during the first day of the strike and decreased dramatically by day 3 (September 12), suggesting that concerns and challenges can vary over the duration of an unplanned school closure. This variability could reflect that parents were initially unprepared for the closure but were able to adapt rapidly to the unanticipated disruption (including help from information shared on social media about access to services). The decrease could also indicate that the public quickly experienced information saturation with a subsequent decrease in interest . Similar trends in social media were identified before, during, and after the March 2011 Japanese earthquake and tsunami . The lack of a perceived direct benefit from the strike may have partially contributed to the consistently negative sentiment score (mean score was -0.61). In general, households are more likely to support unplanned school closures when there is a perceived benefit to the health and wellbeing of the student, regardless of whether the closure is implemented pre-emptively (prior to widespread disease transmission) or reactively (after a larger percentage of students and teachers are already ill) to an infectious disease outbreak [8, 28–30]. We also hypothesize that negative sentiment is more likely to be expressed on social media than positive or neutral sentiment. Social media could serve as an outlet for expressing frustration by a certain but small segment of the population. Background information on the impacted population, such as the percentage of students belonging to single parent households or eligible for free or reduced priced school meals, therefore, could be helpful in interpreting social media findings.
During an influenza pandemic, public health officials will need to rapidly capture and respond to perceived challenges in the implementation of mitigation strategies, including school closures. We explored the use of Amazon Mechanical Turk’s online workforce as a way to rapidly and accurately review and categorize social media posts. To ensure quality results, we requested “master workers” residing in the United States who had a previous rating of 99% or greater on previous projects. This improved the likelihood that Turk workers would be familiar with school closures and related challenges for families and students in the United States, as well as the possible use of American slang and language nuances. On average, five different Turk workers were able to complete the review of each batch within 2–3 hours with relatively good accuracy (>95% for both relevant and irrelevant posts). The number of unique workers participating in the review supports the scalability of this approach. Using such a workforce during a pandemic could compare favorably to household surveys that typically require weeks to months for researchers to collect, clean, and analyze available data. Although a number of machine learning tools are available for automating this process [16, 31], a combination of approaches will likely be needed due to the complexity of language (e.g., sarcasm), as well as misspellings and grammatical errors in social media [16, 19].
We selected the Chicago teachers’ strike for this initial social media project due to the length of the unplanned school closure and the size of the impacted school district (>400,000 students). Although social media use is higher among young adults and in urban and suburban areas , use appears to be independent of education level and race or ethnicity. Therefore, we anticipated that this closure would elicit a high volume of commentary on social media. We captured over 4,500 social media posts during the project period. Due to the nature of the closure, only 20% of the posts were relevant to assessing the impact of the closure on students and their families. Although many of these relevant posts included re-tweets and shares, we did not have access to the metrics required to estimate the specific frequency. The majority of posts were categorized as irrelevant since they addressed the political nature of the strike, the role of unions, and general concerns regarding the welfare and education systems in Chicago. More posts could be relevant or positive if captured during a public health-related closure, where community comments would likely include the perceived health benefits in addition to the costs and consequences of the closure.
We attempted to increase the percentage of posts meeting our definition of “relevant” by including the terms “breakfast,” “childcare,” “daycare,” “lunch,” and “parent” in the Radian6 search criteria. We primarily included these terms to minimize the amount of “noise” related to the political nature of the closure [13–15]. We propose that the any bias resulting from the inclusion of these terms was limited since only one of the terms was required using Boolean logic. Our findings related to child safety and street violence (which were not search terms) support the use of this approach. We also used exact matching for “breakfast,” “childcare,” “daycare,” “lunch,” or “parent” and proximity for “Strike” and “Chicago”. Relevant posts including abbreviations and misspellings could have been missed (e.g., day care). In a qualitative review of relevant posts, however, a large percentage of childcare-related posts referenced childcare as two words. A number of other challenges were identified in categorizing social media posts, including complex posts referencing both political aspects of the strike and the availability of non-educational services to students during the school closure. Additionally, the combination of context and semantics to assign sentiment can involve a level of human subjectivity. To minimize these challenges, we pilot-tested our categorization and sentiment definitions. However, neither the categorization nor sentiment analysis was likely to be 100% accurate.
Despite increased access to social media, information from these sources is likely to be unrepresentative [13–17]. We could only partially limit the geographic scope of social media users with Radian6, and we were unable to determine whether an author of a post was a student in or had a child attending a Chicago City District-school. Our findings could have inadvertently included posts from persons not directly impacted by the Chicago school closure. Capturing IP addresses from social media users when available could help identify the author’s physical location and further assist public health officials in addressing identified negative impacts where and when they occur [13–18, 23, 26–27].
Social media can provide information about the costs and consequences of an unplanned school closure on students and their families. Social media posts are publicly available and can be captured in near real-time to monitor changes in sentiment over time. Social media can also capture the costs and consequences not identified through traditional approaches. The political nature of the teachers’ strike likely influenced the topics and sentiment expressed in this project. Future projects using social media to assess the impact of a public health-related school closure and to capture information on authors’ physical location would be informative. An online workforce possibly combined with new machine learning tools could further improve the capacity to rapidly identify and interpret relevant social media posts. This approach could help public health officials more effectively monitor and balance the anticipated health benefits with possible costs and consequences of unplanned school closures during a future pandemic.
We thank Dr. Noreen Qualls for her support and guidance during the conceptualization and implementation of this project. We also greatly appreciate Mr. Ronnie Henry’s critical feedback in editing this manuscript.
Disclaimer: The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of CDC.
- Conceptualization: JR JK MO.
- Data curation: JR JK BW MO.
- Formal analysis: JR JK BW AP YZ MO.
- Funding acquisition: JR.
- Investigation: JR JK BW AP MO.
- Methodology: JR JK AP BW YZ MO.
- Project administration: JR MO.
- Resources: JR YZ BW MO.
- Software: JR JK BW AP YZ MO.
- Supervision: JR BW.
- Validation: JR JK BW AP YZ MO.
- Visualization: JR JK MO.
- Writing – original draft: JR JK MO.
- Writing – review & editing: JR JK BW AP YZ MO.
- 1. Emergency preparedness and response: school dismissals to reduce transmission of pandemic influenza (abbreviated). www.thecommunityguide.org/emergencypreparedness/schooldismissals.html. Last accessed on June 10, 2015.
- 2. Copeland DL, Basurto-Davila R, Chung W, Kurian A, Fishbein DB, Szymanowski P, et al (2013) Effectiveness of a school district closure for pandemic influenza A (H1N1) on acute respiratory illnesses in the community: a natural experiment. Clin Infect Dis 56:509–16. pmid:23087391
- 3. Chao DL, Halloran ME, Longini IM (2010) School opening dates predict pandemic influenza A(H1N1) outbreaks in the United States. J Infect Dis 202:877–80. pmid:20704486
- 4. Earn DJ, He D, Loeb MB, Fonseca K, Lee BE, Dushoff J (2012) Effects of school closure on incidence of pandemic influenza in Alberta, Canada. Ann Intern Med 156:173–81 pmid:22312137
- 5. Holloway R, Rasmussen SA, Zaza S, Cox NJ, Jernigan DB (2014) Updated Preparedness and Response Framework for Influenza Pandemics. MMWR Morb Mortal Wkl 63:1–24.
- 6. Epson EE, Zheteyeva YA, Rainey JJ, Gao H, Shi J, Uzicanin A, et al (2015) Evaluation of an Unplanned School Closure in a Colorado District: Implications for Pandemic Influenza Preparedness. Disaster Med Public Health Prep 9:4–8. pmid:25739043
- 7. Russell ES, Zheteyeva Y, Gao H, Shi J, Rainey JJ, Thoroughman D, et al (2016) Reactive School Closure during Increased Influenza-like-Ilness (ILI) Activity in Western Kentucky, 2013; A Field Evaluation of Effect on ILI Incidence and Economic Social Consequences for Families. Open Forum Infectious Diseases 3(3):
- 8. Centers for Disease Control and Prevention (2009). Impact of Seasonal Influenza School Closures on Families—Southeastern Kentucky, February 2008. MMWR Morb Mortal Wkl 58:1405–1409.
- 9. Gift TL, Palekar RS, Sodha SV, Kent CK, Fagan RP, et al (2010) Household Effects of School Closure during Pandemic (H1N1) 2009, Pennsylvania, USA. Emerg Infect Dis 16:1315–7. pmid:20678335
- 10. Johnson AJ, Moore ZS, Edelson PJ, Kinnane L, Davies M, et al (2008) Household Responses to School Closure Resulting from Outbreak of Influenza B, North Carolina. Emerg Infect Dis 14:1024–1030. pmid:18598620
- 11. Miller JC, Danon L, O'Hagan JJ, Goldstein E, Lajous M, Lipsitch M (2010) Student Behavior during a School Closure Caused by Pandemic Influenza A/H1N1. PLoS One 5(5): e10425. pmid:20463960
- 12. Centers for Disease Control and Prevention (2010) Parental attitudes and experiences during school dismissals related to 2009 Influenza A (H1N1)—United States, 2009. MMWR Morb Mortal Wkl 59:1131–4.
- 13. Eysenbach G (2009) Infodemiology and Infoveillance: Framework for an Emerging Set of Public Health Informatics Methods to Analyze Search, Communication and Publication Behavior on the Internet. J Med Internet Res 11(1):e11. pmid:19329408
- 14. Salathé M, Bengtsson L, Bodnar TJ, Brewer DD, Brownstein JS, Buckee C, et al (2012) Digital epidemiology. PLoS Comput Biol 8(7):e1002616. pmid:22844241
- 15. Brownstein JS, Freifeld CC, Madoff LC (2009) Digital disease detection-harnessing the Web for public health surveillance. N Engl J Med 360:2153–2155. pmid:19423867
- 16. Salathé M, Khandelwal S (2001) Assessing vaccination sentiments with online social media: Implications for infectious disease dynamics and control. PLoS Computational Biology 7(10):e1002199.
- 17. Signorini A, Segre AM, Polgreen PM (2011) The use of Twitter to track levels of disease activity and public concern in the U.S. during the influenza A H1N1 pandemic. PLoS One 6(5):e19467. pmid:21573238
- 18. Broniatowski DA, Paul MJ. Dredze M (2013) National and location influenza surveillance through Twitter: an analysis of the 2012–2013 influenza epidemic. PloS One 8(12):e83672. pmid:24349542
- 19. Ramagopalan S, Wasiak R, Cox AP (2014) Using Twitter to investigate opinions about multiple sclerosis treatments: a descriptive, exploratory study. F1000Research 3:1–9.
- 20. Freifeld CC, Brownstein JS, Menone CM, Bao W, Filice R, Kass-Hout T, et al (2014) Digital drug safety surveillance: monitoring pharmaceutical products in twitter. Drug Safe 37:343–350.
- 21. Dyer OJ, Castr-Sanchez E, Homes AH (2014) What makes people talk about antibiotics on social media? A retrospective analysis of Twitter use. J Antimicrob Chemother 69:2568–72. pmid:24862092
- 22. Pew Research Center, 2014. Available at: http://www.pewinternet.org/fact-sheets/social-networking-fact-sheet/. Last accessed on June 23, 2015.
- 23. Keller M, Blench M, Tolentino H, Freifeld CC, Mandl KD, Mawudeku A, et al (2009) Early experiences in real-time automated global infectious disease surveillance using unstructured event-based reports. Emerg Infect Dis 15(5):689–95.
- 24. Harris JR, Moreland-Russel S, Choucair B, Mansour R, Staub M, Simmons K (2014) Tweeting for and against public health policy: response to the Chicago Department of Public Health's electronic cigarette Twitter campaign. J Med Internet Res 16(10):e238. pmid:25320863
- 25. Chicago Public Schools. School Data. Demographics-School Year 2012–2013. Available at: http://cps.edu/SchoolData/Pages/SchoolData.aspx. Last Accessed on June 10, 2015.
- 26. Salathé M, Freifeld CC, Mekaru SR, Tomasulo AF, Brownstein JS (2013) Influenza A (H7N9) and the importance of digital epidemiology. N Engl J Med 369(5):401–4. pmid:23822655
- 27. Black DR, Dietz JE, Stirratt AA, Coster DC (2015) Do Social Media have a place in public health emergency response? J Emerg Manag 13:217–26. pmid:26150365
- 28. Kiviniemi MT, Ram PK, Kozlowski LT, Smith KM (2011) Perceptions of and willingness to engage in public health precautions to prevent 2009 H1N1 influenza transmission. BMC Public Health 11:152. pmid:21385436
- 29. Agolory SG, Barbot O, Averhoff F, Weiss D, Wilson E, Egger J, et al (2013) Implementation of Non-Pharmaceutical Interventions by New York City Public Schools to Prevent 2009 Influenza A. PLoS One 8(1):e50916. pmid:23341877
- 30. Dooyema CA, Copeland D, Sinclair JR, Shi J, Wilkins M, Wells E, et al (2014) Factors influencing school closure and dismissal decisions: influenza A (H1N1), Michigan 2009. J Sch Health 84:56–62. pmid:24320153
- 31. Tuarob S, Tucker CS, Salathé M, Ram N (2014) An ensemble heterogeneous classification methodology for discovering health-related knowledge in social media messages. J Biomedical Informatics 49:255–268. pmid:24642081