The objective of Nutrimedia is to evaluate, based on the scientific evidence, the veracity of nutrition claims disseminated to the public by the media. In this article, we describe the methodology, characteristics and contents of this web-based resource, as well as its web traffic and media impact since it was launched. Nutrimedia uses a systematic process to evaluate common beliefs, claims from newspapers and advertising identified and selected by its research team, as well as questions from the public. After formulating a structured question for each claim, we conduct a pragmatic search, prioritizing guidelines and/or systematic reviews. We evaluate the certainty of the evidence using the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach, and classify the veracity of each claim into seven categories (true, probably true, possibly true, possibly false, probably false, false, and uncertain). For each evaluation, we develop a scientific report, a plain language summary, a summary of findings table, and, in some cases, a video. From November 2017 to May 2019, we published 30 evaluations (21 were related to foods, six to diets, and three to supplements), most of which were triggered by questions from the public (40%; 12/30). Overall, nearly half of the claims were classified as uncertain (47%; 14/30). Nutrimedia received 47,265 visitors, with a total of 181,360 pages viewed. The project and its results were reported in 84 written media and 386 websites from Spain and 14 other countries, mostly from Latin America. To our knowledge, Nutrimedia is the first web-based resource for the public that evaluates the certainty of evidence and the veracity of nutrition claims using the GRADE approach. The scientific rigor combined with the use of friendly presentation formats are distinctive features of this resource, developed to help the public to make informed choices about nutrition.
Citation: Rabassa M, Alonso-Coello P, Casino G (2020) Nutrimedia: A novel web-based resource for the general public that evaluates the veracity of nutrition claims using the GRADE approach. PLoS ONE 15(4): e0232393. https://doi.org/10.1371/journal.pone.0232393
Editor: David Meyre, McMaster University, CANADA
Received: November 29, 2019; Accepted: April 9, 2020; Published: April 30, 2020
Copyright: © 2020 Rabassa et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All the results of the evaluations are available in the corresponding technical reports freely accessible on the Nutrimedia website (https://www.upf.edu/web/nutrimedia); all referenced papers are available in the usual databases (MedLine, Cochrane Database of Systematic reviews, etc.). Data of the media impact from Factiva can be reproduced by a search with the term "Nutrimedia". Data on web traffic from Google Analytics is not legally or ethically restricted and it is not necessary to replicate the results –it changes constantly with time.
Funding: Nutrimedia has been partially supported by the Spanish Foundation for Science and Technology (FECYT) grants from the Spanish Ministry of Science, Innovation and Universities (FCT-16-11294 and FCT-17-12460). The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: CPG, Clinical Practice Guideline; GRADE, Grading of Recommendations, Assessment, Development and Evaluation; PICO, Participant, Intervention, Comparison and Outcome(s); RCT, Randomized controlled trial; SoF, Summary of Findings; SR, Systematic Review
Most common chronic diseases such as cardiovascular diseases, cancers and diabetes are leading causes of death, accounting for 71% of all deaths and 43% of the global burden . Modifiable behavioural risk factors such as poor eating or food habits increase the risk of chronic diseases . Raising awareness of scientific knowledge about food and nutrition among the public can help improving health overall, and prevent chronic diseases .
The dissemination of nutrition information in the media is a potential mean for promoting knowledge about appropriate food choices [4–5]. Media (including radio, television, newspapers, internet and social media) today include a wealth of information about food and nutrition. For example, a recent Google search showed over half a million results related to the terms “nutrition advice”. Media has been shown to have a potential impact on knowledge and awareness of the public of health issues related to the field of nutrition [6–7]. However, this information is often misleading and contradictory [8–10]. According to recent research, the public is regularly exposed to mediocre or poor quality information about nutrition on websites  as well as in newspapers . Approximately two-thirds of newspapers related to dietary advice are based on low-quality scientific evidence .
Online resources or websites (e.g., Google searches, YouTube) were the most popular source of nutrition information, between 2003 and 2018, among adults [11–12]. However, to the best of our knowledge, there are no online resources that formally evaluate and communicate the veracity of contemporary claims about nutrition based on scientific evidence. Therefore, we have developed a nutrition web-based information resource for the general public named Nutrimedia that provides rigorous evaluations of claims about nutrition. In this article, we describe how we have developed this resource and present an overview of its characteristics, contents and media impact. The research protocol is available in Spanish from the authors upon request.
Identification and selection of nutrition claims
To identify claims, we developed a preliminary list of claims based on news from the media, Google searches of current topics in nutrition and our knowledge and expertise. This preliminary list has been updated over time.
We classified the topics identified in the following four types of claims:
- Common beliefs related to the health effects of certain diets, foods or nutrients.
- Claims in pieces of news published in newspapers that report about nutrition or nutrition research (newspaper claims).
- Advertising claims in traditional or online media (newspapers, magazines, radio, television, websites, blogs and social networks) about the effect of a particular food (or an ingredient) on consumer health or performance.
- Claims based on questions from the public collected via an online survey posted on Nutrimedia website (e.g., the question "Is meat carcinogenic?" becomes the claim "meat is carcinogenic").
Common beliefs and claims from newspapers or advertising were selected from the preliminary list based on two criteria: 1) achieving the highest interest score (we scored on a 5-point Likert scale the interest of each claim), and 2) ensuring a balanced selection of different types of claims and foods.
To collect questions from the public, we published an online survey from November 20, 2017, until May 4, 2018. The survey included 10 closed-ended questions selected from the preliminary list of claims (to be rated by users on a 5-point Likert scale from 1–5; definitely not to definitely yes interested), one open-ended question and space for comments and suggestions. The questions from the public that were evaluated were selected from those most highly rated among users in the 10 closed-ended questions and those posed in the open-ended question (the selection was made considering interest and feasibility).
Scientific evaluation of nutrition claims
For each claim selected, we developed a scientific report using a systematic and explicit process shown in Fig 1 that includes:
1. Formulation of structured clinical question.
For each nutrition claim, we formulated a structured clinical question in a PICO (participant, intervention, comparison and outcome(s)) format. We included a maximum of four key public-important outcomes (e.g., death, cancer or cardiovascular events).
2. Identification and selection of the evidence.
To identify and select the best available scientific evidence, we conducted a search prioritizing clinical practice guidelines (CPG) and systematic reviews (SR). We searched on electronic databases (e.g., we searched in MEDLINE and Cochrane Database of Systematic Reviews for SR) (S1 Table), combining at least one MeSH term and free-text terms, with title and/or abstract restrictions, for each PICO component. We also searched grey literature (e.g. Google Scholar), governmental and institutional sources (e.g. World Health Organization, U.S. Department of Agriculture, Spanish Agency for Food Safety and Nutrition) and scientific societies websites (e.g., Sociedad Española de Nutrición Comunitaria).
We included CPGs and/or SRs that used systematic methods to search and identify the evidence in two or more databases (e.g., MEDLINE, EMBASE) and that evaluated the risk of bias of research evidence . When we obtained more than one CPG or SR of similar quality we prioritized the most recent. If not available, we included primary studies prioritising randomized clinical trials.
3. Synthesis of the evidence.
We summarized the following information of each of the documents selected: 1) objectives, 2) methods, 3) main results, and 4) conclusion of the authors (if applicable) (S2 Table).
4. Evaluation of the certainty of the evidence.
We assessed the certainty of the body of evidence (CPG or SR or primary studies) about the effects of nutrition interventions using the GRADE approach . GRADE classifies the certainty of evidence as high, moderate, low, or very low for each outcome of interest within the same clinical question. GRADE categorizes randomized controlled trials (RCTs) as high certainty, whereas observational studies start as low certainty. An available body of RCTs can be downgraded on the basis of five factors: study limitations (risk of bias), indirectness, imprecision, inconsistency of results, and publication bias. In the case of observational studies, one can consider rating up on the basis of three factors: large magnitude of an effect, dose-response gradient, and plausible residual confounding (Table 1).
For each evaluation, we generated a Summary of Findings (SoF) table using GRADEPro . SoF tables provide a structured outline of the number of studies and number of participants included for each outcome of interest, certainty (or confidence) of the evidence, and the results both in relative and absolute terms (S3 Table). If the certainty of evidence differed across critical outcomes, the overall certainty of evidence is the lowest certainty of any of the public-important outcomes .
We classified the veracity of each claim according to the certainty of the evidence into seven categories: true, probably true, possibly true, possibly false, probably false, false, and uncertain. For example, we assigned true or false, probably true or false, possibly true or false and uncertain conclusion when the overall certainty of the evidence related to a claim was high, moderate, low and very low (or no studies), respectively (Table 2).
Development of web content for the evaluated nutrition claims
For each claim, we developed the web content in plain language and we structured it in layers and sections, combining text with other formats (images, static and interactive tables, symbols, videos in some cases, and others), with the aim of making it more user-friendly and understandable. We included the following sections in each evaluation :
- Headline: it summarizes the result of the evaluation in a sentence or, in some cases, it poses the question of the evaluation in plain language.
- Introduction and contextualization: we provided a brief introduction and contextualization about the claim being analysed.
- Conclusion: we stated the result of the evaluation with its corresponding symbol.
- Summary: we summarized the evaluation in plain language making the full scientific report available in PDF format. For some claims, we also produced a short video explaining the evaluation .
- What does the evidence say? (Qué dice la ciencia): this section provides a reasoned explanation of the certainty of the research results. When the result of the evaluation was not uncertain, an interactive SoF table was also provided. SoF tables are user-friendly formats to communicate research findings to the public and other stakeholders .
- To know more (Para saber más): this section includes relevant resources and links related to the evaluated claim.
The Nutrimedia website (https://www.upf.edu/web/nutrimedia) is responsive and multi-layered. It is hosted by Pompeu Fabra University, which provided the technical support for its development and maintenance.
Media impact and website traffic analysis
We used two main methods to assess the media impact of Nutrimedia: the Factiva database, which includes newspapers, magazines and news agencies from all over the world; and the media monitoring service Acceso 360. The estimated economic value of each piece of news, provided by Acceso 360, was calculated based on the cost of the advertising space it occupies in a particular newspaper. We used Google Analytics to analyse website traffic.
Nutrimedia was launched in Spanish in November 20, 2017. During the first year and a half, we published 30 evaluations, of which 12 (40%) were questions from the public, 9 (30%) were common beliefs, 7 (23%) were newspaper claims, and 2 (7%) were advertising claims. Most evaluations were related to raw or processed foods (70%; 21/30), followed by diets (20%; 6/30) and supplements (10%; 3/30); and the majority of them were based on SR (80%; 24/30). Nearly half of the claims were classified as “uncertain” (47%; 14/30), followed by “possibly true” (13%; 4/30), probably true (13%; 4/30), “probably false” (10%; 3/30), “possibly false” (7%; 2/30), false (7%; 2/30) and “true” (3%; 1/30) (Table 3). An example of the evaluation of the veracity of a claim is presented in the S4 Table.
Nutrimedia has other contents such as “Eating with science” (Comer con ciencia), “To know more” (Para saber más), and “About Nutrimedia” (Sobre Nutrimedia). The “To know more” section provides short videos to promote critical thinking and to facilitate the understanding of some methodological concepts (e.g., what is the GRADE approach and how certainty of the evidence was evaluated; guidelines for the public to interpret information on nutrition; guidelines for journalists to report on nutrition; relevant links in the field of nutrition, and a glossary). In the section “About Nutrimedia”, users can find out who we are, the scientific methodology that we applied to evaluate claims on nutrition, the press releases and the media coverage.
In the section “Ask Nutrimedia”, available from November 20, 2017, to May 5, 2019, the public assessed the interest of 10 questions in an online survey. Of the 12 questions from the public that were evaluated, 6 were the most voted of the 10 closed-ended questions and 6 were posed by users in the survey space for comments and suggestions. Some other questions about nutrition of general interest, most of them posed by the public, were answered by experts in articles and interviews (provided as podcasts) in the “Eating with science” section. The number of online survey respondents was 333 (55.2%, 182/333, general public; and 45.8%, 151/333, health professionals). Fifty-eight respondents (17.4%, 58/333) provided positive (51.7%, 30/58) (e.g., “An excellent idea to combat all the misinformation”; “Thank you for helping me get back to believing in science”) and neutral (5.2%, 3/58%) comments or suggestions (43.1%, 25/58) (e.g., “It would be interesting to receive web updates by email”; “It would be tremendously helpful to give healthy and tasty recipes”).
The number of website users was 47,265 (46,052 new users) since the website was launched (November 20, 2017) and for the first 18 months (until May 20, 2019). The number of page views in this period was 181,360. During this period, Nutrimedia was cited by 94 newspapers, news agencies and newswires included in Factiva Dow Jones database; of these 94 citations, 55 were in Spanish, 33 in Catalan, 4 in English and 2 in Portuguese. According to the data provided by Acceso 360, Nutrimedia was cited 78 times by newspapers and 386 times by websites from Spain and 14 other countries, mostly in Latin America. In addition, the Nutrimedia project and its evaluations were broadcasted on several Spanish radio and television channels. In that period, the estimated economic value of the 78 pieces of news published in printed editions of newspapers was €152,507.55.
Nutrimedia has also been cited by prominent Spanish dieticians and nutritionists. For example, Nutrimedia is referenced in 5 of its 12 recommendations for the general public in the 2018 food-based dietary guideline "Small changes to eating better" by the Public Health Agency of Catalonia .
Finally, Nutrimedia was well placed in nutrition searches in Spanish with Google, whose algorithm rates the pages according to the criteria of expertise, authority and trustworthiness ; e.g., in a Google search with the terms “meat cancer” (search performed on 2019 September 30, using a logged-out Chrome browser cleared of cookies and previous search history), Nutrimedia was ranked as the third listed information source in a list headed by the World Health Organization.
Nutrimedia is the first web-based resource for the general public that evaluates the veracity of media claims about nutrition based on the certainty of evidence, and to communicate the results in plain language and friendly presentation formats. We have evaluated 30 claims and classified nearly half of them as uncertain based on GRADE. During the first year and a half, Nutrimedia had a considerable impact on the media (about one impact per day).
There are some other online resources with different purposes and target populations. “Practice-based evidence in nutrition” provides the latest evidence in practice-based nutrition questions applying the GRADE approach, but this evidence-based decision support system is intended for dieticians/nutritionists and students . “Behind the Headlines”, from the UK National Health Service, analyses critically media claims related to nutrition, but does not assess the certainty of the evidence . Cochrane Nutrition provides nutrition-related Cochrane systematic reviews, but there is often a communication gap between these reviews and the public . To avoid this, Cochrane centers implement alternative ways to make evidence accessible in plain language summaries  and blogshots (infographics that summarize the evidence) in English , Spanish  and other languages. By overcoming these limitations, Nutrimedia is an innovative resource for disseminating quality nutrition information, and making it accessible to the public. Therefore, we believe that Nutrimedia is a pioneering initiative that promotes critical thinking and can have an impact on the food and nutrition literacy of the general public .
Nutrimedia has several strengths. The website is user-friendly because no registration is required, the evaluations are accessible within a few clicks and its content is multi-layered and multiformat. For each evaluation, we have applied a rigorous and explicit methodology using GRADE. GRADE approach represents a systematic, explicit and transparent methodological framework for grading the certainty of evidence and it has already been endorsed or used by over 100 organizations, the World Health Organization and the Cochrane Collaboration [28–29] among others. Finally, our project team has extensive knowledge and experience in the fields of nutrition, evidence-based medicine, methods, communication and journalism.
Nutrimedia also has some limitations. Firstly, it is only available in Spanish. Secondly, we used pragmatic search strategies favoring precision over sensitivity . Thirdly, this project has no system yet for monitoring and continuous updating of the evaluations. Finally, analysis of its impact, usefulness, accessibility and understandability is still limited.
Implications for practice and research
General public, journalists and communicators can use Nutrimedia to stay informed and making informed decisions about nutrition. Researchers interested in evaluating topics about nutrition can use our approach. More research is needed about the impact, usefulness, accessibility and understandability of Nutrimedia.
S2 Table. Key reporting aspects of the evidence.
S3 Table. Standardised statements about effect according to the GRADE approach.
We would like to thank Darío Lopez, Andrea Juliana Sanabria, Mónica Ballesteros, Carolina Requeijo Lorenzo, Karla Salas Gama, Paulina Fuentes, Laura Martínez García, and Alba Irigoyen for their contribution in this project. We would also like to thank to Victoria Leo for her assistance with the English text.
- 1. World Health Organization (WHO). Noncommunicable diseases. Key facts. 2018 June 1 [cited 2019 Oct 25]. In: WHO website [Internet]. https://www.who.int/news-room/fact-sheets/detail/noncommunicable-diseases
- 2. GBD 2017 Diet Collaborators. Health effects of dietary risks in 195 countries, 1990–2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet. 2019; 393: 1958–1972. pmid:30954305
- 3. Mozaffarian D. Dietary and policy priorities for cardiovascular disease, diabetes, and obesity: A comprehensive review. Circulation. 2016;133: 187–225. pmid:26746178
- 4. Helm J, Jones RM. Practice Paper of the Academy of Nutrition and Dietetics: Social Media and the Dietetics Practitioner: Opportunities, Challenges, and Best Practices. J Acad Nutr Diet. 2016;116: 1825–1835. pmid:27788767
- 5. Viswanath K, Bond K. Social determinants and nutrition: reflections on the role of communication. J Nutr Educ Behav. 2007;39: S20–4. pmid:17336801
- 6. Nagler RH. Adverse outcomes associated with media exposure to contradictory nutrition messages. J Health Commun. 2014;19: 24–40. pmid:24117281
- 7. Williams G, Hamm MP, Shulhan J, Vandermeer B, Hartling L. Social media interventions for diet and exercise behaviours: a systematic review and meta-analysis of randomised controlled trials. BMJ Open. 2014;4: e003926. pmid:24525388
- 8. Gholizadeh Z, Papi A, Ashrafi-Rizi H, Shahrzadi L, Hasanzadeh A. Quality evaluation of Persian nutrition and diet therapy websites. J Educ Health Promot. 2017;6: 48. pmid:28616415
- 9. Kininmonth AR, Jamil N, Almatrouk N, Evans CEL. Quality assessment of nutrition coverage in the media: a 6-week survey of five popular UK newspapers. BMJ Open. 2017;7: e014633. pmid:29284712
- 10. Cooper BE, Lee WE, Goldacre BM, Sanders TA. The quality of the evidence for dietary advice given in UK national newspapers. Public Underst Sci. 2012;21: 664–73. pmid:23832153
- 11. Quaidoo EY, Ohemeng A, Amankwah-Poku M. Sources of nutrition information and level of nutrition knowledge among young adults in the Accra metropolis. BMC Public Health. 2018;18: 1323. pmid:30497442
- 12. Ostry A, Young ML, Hughes M. The quality of nutritional information available on popular websites: A content analysis. Health Educ Res. 2008;23: 648–55. pmid:17897928
- 13. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0. 2019 August [cited 2019 Oct 25]. In: Cochrane website [Internet]. www.training.cochrane.org/handbook
- 14. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336: 924–6. pmid:18436948
- 15. GRADE working group. Windows-based GRADEpro 3.0 software. 2015 [cited 2019 Oct 25]. In: GRADEpro GDT website [Internet]. http://gdt.guidelinedevelopment.org/app/
- 16. Guyatt GH, Oxman AD, Santesso N, Helfand M, Vist G, Kunz R, et al. GRADE guidelines: 12. Preparing summary of findings tables-binary outcomes. J Clin Epidemiol. 2013;66: 158–72. pmid:22609141
- 17. Nutrimedia. Mensajes Evaluados. ¿Es realmente cancerígena la carne? 2019 [cited 2 March 2020]. In: Nutrimedia website [Internet]. https://www.upf.edu/web/nutrimedia/-/comer-carne-puede-elevar-ligeramente-el-riesgo-de-cancer-pero-este-riesgo-es-muy-inferior-al-de-fumar-o-consumir-alcohol#.Xlzh5df0nbi
- 18. Nutrimedia. Para saber más. Vídeos para el pensamiento crítico. 2019 [cited 2 March 2020]. In: Nutrimedia website [Internet]. https://www.upf.edu/web/nutrimedia/videos-para-el-pensamiento-critico
- 19. Rosenbaum SE, Glenton C, Oxman AD. Summary-of-findings tables in Cochrane reviews improved understanding and rapid retrieval of key information. J Clin Epidemiol 2010;63: 620e6
- 20. Public Health Agency of Catalonia. Health Department. “Small changes to eat better” [Internet]. Public Health Agency of Catalonia. 2018 Nov. [cited 2019 Oct 25]. http://salutpublica.gencat.cat/web/.content/minisite/aspcat/promocio_salut/alimentacio_saludable/02Publicacions/pub_alim_salu_tothom/Petits-canvis/La-guia-peq-cambios-castella.pdf
- 21. Google. Search quality general guidelines. 2019 Dec 5 [cited 2020 Feb 24]. In: Google [Internet]. https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf
- 22. Practice-Based Evidence in Nutrition (PEN). The global resource for nutrition practice. 2019 [cited 2019 Oct 25]. In: PEN website [Internet]. http://www.pennutrition.com/index.aspx
- 23. National Health Service (NHS). Behind the headlines: food and diet. 2019 [cited 2019 Oct 25]. In: NHS website [Internet]. https://www.nhs.uk/news/food-and-diet/?order=newest
- 24. Cochrane Nutrition. Evidence. Cochrane nutrition reviews. 2019 [cited 2019 Oct 25]. In: Cochrane Nutrition website [Internet]. https://nutrition.cochrane.org/evidence
- 25. Cochrane UK. Blogshot & infographic archive. 2019 [cited 2019 Oct 25]. In: Cochrane UK website [Internet]. http://uk.cochrane.org/blogshot-infographic-archive
- 26. Cochrane Iberoamérica. Blogshots. 2019 [cited 2019 Oct 25]. In: Cochrane Iberoamérica website [Internet]. http://es.cochrane.org/blogshots_temas#n
- 27. Yuen EYN, Thomson M, Gardiner H. Measuring Nutrition and Food Literacy in Adults: A Systematic Review and Appraisal of Existing Measurement Tools. Health Lit Res Pract. 2018;2: e134–e160. pmid:31294289
- 28. Alonso-Coello P, Schunemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. [GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction]. Gac Sanit. 2018;32: 166.e1–e10.
- 29. Alonso-Coello P, Rigau D, Sanabria AJ, Plaza V, Miravitlles M, Martinez L. Quality and strength: the GRADE system for formulating recommendations in clinical practice guidelines. Arch Bronconeumol. 2013;49: 261–7. pmid:23434203
- 30. Martinez Garcia L, Sanabria AJ, Araya I, Lawson J, Sola I, Vernooij RW, et al. Efficiency of pragmatic search strategies to update clinical guidelines recommendations. BMC Med Res Methodol. 2015;15: 57. pmid:26227021