Examining the availability/findability of stimuli employed in social media and body image research

Since 2012, and the start of the ‘replication crisis’ [1], Psychology researchers (as well as researchers in other disciplines, e.g., biology; see [2]) have increasingly engaged in a set of ‘open research’ practices. ‘Open research’ practices include the pre-registration of predictions and analysis plans (e.g., [3]), the use of open-source software (e.g., [4]), and the open sharing of data (e.g., [5]). By adopting these practices, individuals aim to increase the transparency and rigor of their research. As a result, the likelihood that they will generate robust, replicable findings should be increased.

A further key ‘open research’ practice is the sharing of research materials, such as questionnaires and tasks. Reproducing others’ work is a cornerstone of science [6] and researchers who openly share their research materials facilitate this core process [7]. However, it appears that the norm across many disciplines is to not openly share research materials. For example, in a recent study that sampled 250 articles published in Psychology journals between 2014 and 2017, Hardwicke et al. [8] reported that research materials were shared in only 14% of those articles. Similarly, in an analogous study where 250 social science articles (i.e., articles from Psychology, but also from Geography, Economics, and Political Science, for example) published between 2014 and 2017 were examined, 11% were rated as openly sharing their research materials [9].

One particular area within Psychology where the sharing of research materials could be considered to be especially important is in research that examines the impact of images from social media on body image. This evidence base typically examines how exposure to images from social media that, for example, promote the ‘thin ideal’ (e.g., [10]) or that serve as ‘fitspiration’ (e.g., [11]) affect participants’ body image (or related constructs) versus exposure to control images (such as nature scenery). The images used in these studies are clearly very important to the design of the study, as their content should determine the magnitude of the intended manipulation, and so it could be considered especially important that researchers working in this area share the images they have used as task stimuli. However, the experience of two of the authors (AA and DS) in designing a study that examined the effect of exposure to images from social media that promote the ‘thin ideal’ on women aged 18–22 years versus women aged 31–45 years (see: doi.org/10.17605/osf.io/zfjqh), suggested that the sharing of stimuli was very uncommon in this area.

Given this experience, the present study aimed to investigate this issue systematically, by examining how available (or findable) images used as task stimuli in research on social media and body image were. In addition, given that there is some indication that open research practices are becoming more common over time (e.g., [12]), we tested the possibility that there would be an association between year-of-publication and open sharing of the images used as task stimuli.

Method

Results

Discussion

The aim of this study was to examine the availability/findability of the images used as task stimuli in research that tested the effect that exposure to images from social media have on participants’ body image (and related constructs). We found that very few of the articles we coded made the images employed fully available. Many more articles gave only broad, imprecise descriptions of the images they used. Thus, it seems that in experimental research on the impact of social media on body image, there has been little adoption of open sharing of study materials.

These findings are consistent with the results of past research that has examined the availability of tasks/task stimuli in Psychology research. For example, Hardwicke et al. [8] reported that research materials were shared in only 14% of a random sample of Psychology articles published between 2014 and 2017. Similarly, across a random sample of 250 articles from the social sciences published between 2014 and 2017, only 11% openly shared their research materials [9]. Thus, the pattern we have reported here, of the norm in social media and body image research being a failure to share research materials, is also the norm in other areas of Psychology and in other social science disciplines.

The lack of open sharing of the images used as task stimuli in social media and body image research likely has several implications or consequences. First, this makes reproducing others’ work much more difficult, as without certainty around what stimuli were used in a study researchers cannot be confident that they are creating a task that is equivalent to the one used in the original study. Given that reproducing others’ research and attempting to replicate their findings is a cornerstone of the scientific method [6], it is clearly a substantial problem if researchers are not able to do so.

Related to this is a second consequence: the ability of researchers to ‘successfully’ replicate others’ work (i.e., to find the same effects that have been reported in previous studies) is probably reduced by the failure to openly share task stimuli. This is because where a researcher cannot precisely re-create the task used in an original study, and so must source a different set of stimuli when developing their own version of the task for a subsequent study, the magnitude of the effect elicited by the task in the subsequent study will almost certainly differ from the magnitude of the effect elicited by the task in the original study. And so, across studies that use different sets of stimuli, we should expect effect sizes to vary beyond sampling error, possibly to the extent that some studies will find significant effects on body image and other studies will not. Thus, what appears to be inconsistency within an evidence base, may simply be a result of the use of variation in stimuli employed. To some extent, variation in the stimuli used across studies can strengthen an evidence base as it shows whether an effect is generalizable. And, as de Valle et al. [13] noted this was the case in their meta-analysis, where effects were similar across studies that employed different types of images (e.g., images that promote a traditional ‘thin ideal’ image versus images that promote specific forms of ‘fitspiration’). However, if materials were shared openly, it would be easier for researchers to examine the effects of the use of different images in a systematic manner (as Yarkoni [21] recommended).

The third consequence of a lack of sharing of images used as task stimuli in social media and body image research is one of research waste [14]. Without straightforward access to the task stimuli that others have used previously, researchers will unnecessarily spend a (presumably) substantial amount of time sourcing their own stimuli, building a task using this stimuli, and (ideally) piloting this task to ensure it works as intended. At present, there are no good estimates of how much time researchers spend unnecessarily developing tasks that could have been shared openly. It would be useful for future research to provide these kind of estimates, as has been done for estimating the cost of conducting a meta-analysis (around $141,000; [22]), and the number of ‘redundant’ meta-analyses that are published (possibly more than 50% of all published meta-analyses; [23]).

It is likely that there are a number of systemic factors that have contributed to the lack of sharing of images used in social media and body image research. For older studies in our sample, it is likely that sharing task stimuli was both not something that researchers were actively encouraged to do and was something that was relatively complex to achieve. For example, the Open Science Framework, which many researchers now use to share data and research materials [24], was only established in 2013. Prior to its existence, researchers may not have had a good understanding of which online repositories would be effective places to share study materials from. More recently, it is likely that a simple lack of resources/time-pressure discourage researchers from sharing task stimuli.

However, another issue may be that researchers are hesitant to share tasks that feature stimuli taken from social media because of concerns about privacy or copyright. That is, they may worry that if they develop a task that contains images from a celebrity’s social media account and then share that task freely online, they may be violating copyright law. Such concerns are warranted in some contexts (see [25]), but will not be relevant for many researchers, as we explain below.

Concerns about privacy are very much warranted where researchers have used images (or other types of content, such as text) posted to ‘private’ accounts, or content that the original poster intended to be private to some extent. See, for example, Morant et al. [26], where content from Twitter was used by researchers in a way that was upsetting for the original Twitter users, who felt they had engaged in private conversations, which should not have been used as data by the researchers without consent being obtained. In these contexts, researchers should request permission from the account holder to use the content they have posted for research purposes. If this is provided, then researchers should discuss with the account holder whether they also consent to that content being shared openly with other researchers (e.g., via the Open Science Framework, or in supplementary materials to an article), as it is possible that an account holder may consent to content from their posts being used in a single research project, but may not consent to that content being widely shared.

In contrast, concerns about copyright law are less warranted where researchers have used images posted to ‘public’ social media accounts, which are easily discoverable by anyone accessing a social media platform such as Instagram, Facebook, Twitter, or TikTok. Typically these images can be used under the doctrine of ‘fair use’ (see help.instagram.com/116455299019699?helpref=faq_content and https://copyright.gov/fair-use/), as long as researchers include a citation to accompany the content they have used/shared. A good example of this is Lalancette and Raynauld’s [27] article, which analysed the use of online images in ‘celebrity politics’ in Canada, and re-produced 11 images from Canadian Prime Minister Justin Trudeau’s Instagram account, with each image accompanied by an appropriate citation. Similarly, as part of a research project conducted by AA and DS, we posted PowerPoint files containing the images we employed as stimuli on the Open Science Framework (see https://osf.io/egamu and https://osf.io/yce9z) in a way that does not violate copyright law.

Researchers can raise concerns they have around privacy and copyright with copyright librarians at their institutions, who should be able to provide clear guidance about what best practice is in terms of re-using and sharing content taken from social media accounts. If this advice is not available to a researcher, and they wish to err on the side of caution, then posting the URLs where the images used can be found, as well as a detailed description of the images, is another way in which researchers can make their stimuli available to others (see https://osf.io/y796h and https://osf.io/g7xem for examples of this). However, it should be noted that there are limitations to this approach of sharing URLs, as they may be affected by ‘link rot’ [28], where once-usable links become unavailable over time. This is especially likely to be a problem when trying to provide links to social media accounts, as it is possible to automatically delete posts from social media, once a post is a certain ‘age’ or so that an account does not contain more than a specific number of posts (see tweetdeleter.com/features/auto-delete-tweets and aischedul.com/delete-instagram-posts-automatically-after-publishing/). Thus, best practice likely involves posting both the URLs where the images used as task stimuli can be found, as well as re-productions of the images used, accompanied by appropriate citations.

It is, therefore, possible for researchers – at least in most cases – to share the images from social media that they have used as task stimuli, and there are good reasons (related to increasing reproducibility, increasing replicability, and reducing research waste) to do so. However, here we found no evidence that open sharing of stimuli had increased over time, which suggests that adoption of this practice will not occur without specific interventions. Interventions that may increase the open sharing of stimuli by researchers working on social media and body image include journals implementing policies that mandate the sharing of research materials and/or journals awarding ‘badges’ that acknowledge articles where the authors have shared research materials. Journal policies mandating data-sharing (or the inclusion of a statement explaining why data cannot be shared) have been shown to be associated with a substantial increase in that practice, although it should be noted that data was often shared in sub-optimal ways [29]. The awarding of badges by journals for articles that have adopted ‘open research’ practices, such as the pre-registration of analysis plans and the sharing of analysis code, has also been associated with an increase in the adoption of these practices [30]. The International Journal of Eating Disorders has recently introduced a ‘badges’ system and new editorial policies around open science [7], and research that examines the impact these interventions have on the open sharing of research materials on social media and body image research published in that journal versus social media and body image research published elsewhere will be of interest.

This study suffered from several limitations. First, the sample of articles we coded was relatively small in comparison to other studies that have examined ‘open research’ practices (e.g., samples of 250 papers were used in [8] and [9]). That being said, this study focused on a narrower subset of articles, and so this smaller sample of papers is almost inevitable. In addition, others (e.g., [31]) have used similar (even smaller) samples of articles to investigate the kinds of questions we have examined here. Second, because we employed a sample of papers generated by researchers who conducted a meta-analysis in 2021, the articles we reviewed do not reflect practices researchers have engaged in 2022–2025. Given that we did not find an association between year-of-publication and increased sharing of images used as task stimuli, we would be surprised if there had been a substantial change in this practice in 2022 and 2023. Although, as we noted above, the International Journal of Eating Disorders has recently implemented new policies and incentives around engaging in ‘open science’ practices, and future research should examine if sharing of images used as task stimuli becomes more common in social media and body image research through the mid-2020s. Third, we developed our coding system based on our past experience of reading publications in this field, rather than using coding systems developed by other researchers. This may have introduced some bias in, for example, the codes generated. Fourth, the concerns we have raised about open sharing of materials would be less of a problem if, when contacted, the authors of a publication were able to straightforwardly share (e.g., via private email correspondence) the materials they had used. We took the approach of assuming that contacting authors would have been unproductive, and we think that this is a fair assumption based on, for example, data that around 85% of authors do not respond to requests to share data [32]. However, it is possible that authors would be more willing to discuss sharing of materials than data. Future research that examines how often authors of social media and body image research are willing/able to share testing materials when requested would be valuable.

Finally, another limitation caused by our use of a sample of papers generated by other researchers when they were conducting a meta-analysis is that the value of our study depends, at least to some extent, on the comprehensiveness of their search strategy and the rigor with which it was carried out. Our judgment was that the literature search conducted by de Valle et al. [13] was an effective one, in that the terms employed were broad enough so that relevant studies were discovered and in that several of the criteria by which systematic reviews are evaluated (using more than one database to search for publications, using more than one researcher to make decisions about including/excluding publications, using more than one researcher to extract data from publications; [33]) were met. However, it is true that the limitations that applied to de Valle et al.’s search strategy (e.g., excluding studies that were not written in English) also apply to the current study.

In summary, this study aimed to examine the availability/findability of images used as task stimuli in social media and body image research. In line with research that has examined similar questions in other disciplines, we found that open sharing of task stimuli was rare. Future research that examines whether this changes throughout the 2020s, as open research practices become incentivized by a growing number of journals, will be valuable.