Figure 1.
Flow chart of crowdsourcing procedures.
Steps are separated into sections based on separate tasks. The more granular of a process that can be made, the more amenable the process is to crowdsourcing. MTurk: process completed on MTurk. PI: process completed by AWB. R: process completed with custom R scripts. Gray boxes include tasks automated through R or MTurk, while steps outside of the gray boxes were manually completed. Circled numbers represent the number of times a task was completed. In box E, 304 preliminarily included abstracts (including the 158 ultimately included) had citation counts extracted while the final abstract ratings were concurrently being completed from box A.
Figure 2.
Performance metrics of microworkers.
In each chart, green = desirable, yellow = acceptable, red = undesirable, blue = issue external to crowdsourcing, and black = failure. The letters preceding each chart title corresponds to the steps in Fig. 1, and the number in parentheses represents the number of units compared. A) Two: consensus after two ratings; Three: consensus after three ratings; “Other”: consensus after reviewing free-text answers; MR: manually reviewed by AWB. The accompanying histograms show the distributions of hourly pay calculated by completion time, and completion time in seconds for each task. B.3) Review: microworkers did not agree the foods matched, AWB reviewed the foods extracted from abstracts, and they matched; Fail: the foods did not match after review from AWB. B.12) MR Yes: AWB determined the synthesized term matched the original two foods; MR No: AWB determined the synthesized term did not match the original two foods. C.1) Popular opinion questions were either completely answered or not. C.2) Known: less than 15% did not know the food; Unknown: greater than 15% did not know the food. E) Match: 3 ratings matched; Consensus: 2 of 3 matched; Error: the link provided to the microworkers was faulty.
Figure 3.
Example of groups of foods and average obesogenicity ratings.
Each food listed on the y-axes is shown as synthesized by microworkers in Fig. 1 Step B.9. Foods are ordered within each panel top to bottom with the personal opinion from most to least obesogenic. The vertical dotted line represents the transition from popular opinion indicating the food prevents obesity (left) to causes obesity (right). Each point represents the mean ±95% CI. Note that these results are only meant to reflect an estimate of popular opinion, and do not necessarily reflect the actual obesogenicity of any given food.
Figure 4.
Citations increase with journal quality and time since publication.
To confirm that extracted citation counts, journal rank, and publication date conformed to expected patterns, citation counts were fit as a function of publication date controlling for journal quality (SJR; upper panel) and citation counts were fit as a function of journal quality (lower panel). No differences were seen between papers that were “known” (included) and “unknown” (excluded).
Figure 5.
Quartile-based categorization of perceived obesogenicity foods.
Foods were categorized dependent on the median and quartiles as follows: Good (“Prevents Obesity”), median ≤−1 and Q3≤0; Bad (“Causes Obesity”), median ≥1 and Q1≥0; Not Related, median and IQR between −1 and 1; Mixed (depends on who eats the food), median and IQR spanning other categories. Each dot represents the median spanned by Q1 to Q3 for each included food.