Introduction

Surveys have been instrumental during the COVID-19 pandemic to capture attitudes and (health-related) behavior of the general public [1]. Based on these survey data, researchers were able to derive recommendations for political action [2,3]. Due to the high demand for rapid data collection at the onset of the pandemic, survey researchers in the social sciences were forced to make compromises in a trade-off between high-quality survey design and quick collection of data [1,4–6]. For instance, von Glasenapp et al. [7] and Gummer et al. [8,9] reported that surveys with design features often associated with lower data quality were more frequently used during the onset of the pandemic than high-quality designs. Especially, survey designs utilizing fast-to-implement web mode and less costly, quick-to-access non-probability online access panel samples were frequent during these times. These authors also showed that the survey projects that were quick to collect data and produce findings were less likely to publish data (quickly) and provide comprehensive documentation to facilitate re-use of their research data.

The availability and reusability of research data is key to the scientific process, whether for the purpose of replication or for the efficient use of research funding to produce data as a common good. As computer-assisted tools have become increasingly important for finding and using data, it is essential to enhance data management and stewardship so that it can be easily found, retrieved, integrated, and reused in future research [10–12]. To assess these features of datasets, Wilkinson et al. [12] proposed the FAIR principles that include Findability, Accessibility, Interoperability, and Reusability of research data. Findability means that the data can easily be found by both humans and machines. Accessibility indicates that the data can be further accessed and retrieved after being found. Interoperability indicates that the data is compatible with other data or tools. Reusability means that adequate information about the data is provided so that other researchers can reuse it. The FAIR principles can further be differentiated into 15 subdimensions and can be used as a framework for producing, documenting, and publishing data, as well as to describe and evaluate the FAIRness of existing data.

Previous research on the availability and reusability of survey data has either directly acknowledged and investigated the FAIRness of data or focused on specific aspects without directly making use of the FAIR framework (e.g., content of survey documentations). In the first category, focusing on election studies, Eder and Jedinger [13]reported that most election studies of Western democracies were well-documented and that their level of reusability was high. Adding to this, Zenk-Möltgen [14] showed that the FAIRness of national election data improved between 2018 and 2024. Albeit not only focusing on survey data, the studies by Batzdorfer et al. [15] and Sofi-Mahmudi et al. [16] argue that during the COVID-19 pandemic, the FAIRness of research data (esp. reusability) diminished.

In the second category, the studies by Jabkowski [17], Kołczyńska and Schoene [18], and Stefkovics et al. [19] investigate reporting standards in survey research. Jabkowski [17] focused on the reporting of key information on sampling, fieldwork procedures, outcome rates, and weighting. These pieces of information are central elements of survey documentation and reflect similar concepts as Reusability in FAIR (i.e., they enable data users to understand how data was collected and for which purposes they can be used). Closely related, Kołczyńska and Schoene [18] focused on documentation of sampling, outcome rates, questionnaire development, pretesting, and data quality controls. Adding a more nuanced view on outcome rates, Stefkovics et al. [19] investigated whether and how these were calculated and reported in journal articles using survey data. All three studies show that survey data documentation has improved in recent years. Focusing on the COVID-19 pandemic, von Glasenapp et al. [7] reported that during the onset of the pandemic, survey projects were more likely to focus on publishing first results quickly and provide less timely data access and thorough documentation. Further, these authors showed that data from survey projects that utilized easy and quick-to-implement survey designs (i.e., non-probability web surveys) were less accessible and interpretable compared to other survey projects. The authors concluded that this was the result of a trade-off within survey projects between quality and rapid data collection. Similar findings were reported by Gummer et al. [8], who showed that newly established longitudinal surveys during the COVID-19 pandemic focused on the quick production of results and less so on generating well-documented and accessible data.

Both lines of research have gaps that are detrimental to a better understanding of the FAIRness of survey data during the COVID-19 pandemic. Studies focusing on FAIRness have not specifically examined survey data collected during the COVID-19 pandemic [13–16]. Even when focusing on research data on the COVID-19 pandemic [15,16], it remains an open question how these generalize to social science surveys. Moreover, in this line of research, characteristics of the survey design have been neglected. The latter gap is unfortunate as decades of survey research have made a case that survey design can impact data quality. If FAIRness would differ across surveys with different designs, replication, and transparency would be biased towards those designs with higher FAIRness.

In contrast, studies focusing on surveys have neglected the full scope of the FAIR principles [7,8,17–19] and only a few focused on the COVID-19 pandemic [7,8]. This gap is rather important since previous studies reported that during the pandemic, the need for producing data quickly might have pushed researchers to make decisions that resulted in less well-documented data [7,8], possibly interrupting a positive trend in more comprehensive survey documentations [17–19]. Further, studies in this line of research mostly have a rather narrow perspective and focus on elements of the study or methodological documentation [7,8,17–19] or limit their perspective to the general availability of data [7,8]. As before, with the exception of the most recent studies [7,8], this line of research also largely neglects differences across survey designs. Again, this research gap is unfortunate as transparency and replication would be especially important for surveys, which, due to their design, raise questions regarding possible quality issues—possibly introducing biased replication and accessibility potentials. To summarize, in previous research, we see research gaps with respect to the intersection of (i) investigating the full scope of the FAIR principles, (ii) considering the important time during the COVID-19 pandemic, and (iii) acknowledging differences in FAIRness across survey design.

With the present study, we aim to address the gaps in prior research for surveys collected during the COVID-19 pandemic in Germany. We selected Germany as a case because it is characterized by an advanced research landscape with well-developed research infrastructures. Among researchers, a general knowledge about the need for replicability and transparency can be expected, and infrastructures for research data management exist. Moreover, focusing on surveys conducted in the same country allows us to keep the country context of each survey comparable. Consequently, we address the following research questions (RQs) on the FAIRness of survey data collected during the COVID-19 pandemic in Germany:

RQ1. How FAIR are survey data collected during the COVID-19 pandemic in Germany?

RQ2. Does the FAIRness of survey data collected during the COVID-19 pandemic in Germany differ by survey design?

To answer our research questions, we utilized data collected as part of the “Survey Data Collection during the COVID-19 Pandemic” (SDCCP) project [20] that includes metadata on 717 surveys collected during the COVID-19 pandemic in Germany [21]. We assessed FAIRness scores for those surveys published under a persistent identifier (i.e., DOI) or a project website utilizing the F-UJI assessment tool [22,23].

In the next section, we detail our data and methods. After presenting our results, we offer concluding remarks and outline opportunities for future research.

Data and methods

Analyses

In our analyses, we used data restricted to unique identifiers (N = 319). For this purpose, we excluded duplicate cases that shared a DOI or URL. For duplicates, we included only the earliest entry in the dataset. Fig 1 illustrates our exclusion process. Excluding duplicate cases was necessary as the SDCCP included multiple entries for longitudinal survey programs (see above) and would otherwise overrepresent these surveys.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Flowchart of the Sample Selection.

https://doi.org/10.1371/journal.pone.0358755.g001

To answer RQ1 (“How FAIR are survey data collected during the COVID-19 pandemic in Germany?”), we provided descriptive statistics of the FAIRness measures we obtained for the surveys. We complemented total FAIRness scores with FAIRness of subdimensions for a more comprehensive description.

To answer RQ2 (“Does the FAIRness of survey data collected during the COVID-19 pandemic in Germany differ by survey design?”), we used descriptive statistics conditional on sampling method and survey mode. In addition, we ran an Ordinary Least Squares (OLS) regression model with the overall FAIRness as the dependent variable and the combined survey design variable as the independent variable. For additional insights, we reran the model twice with the sampling method and mode as independent variables instead. We decided to use OLS due to its general robustness and easy accessibility of results to readers (for a robustness check of the model choice, see below).

Robustness checks. We performed a series of robustness checks. First, we investigated whether the existence of a DOI (instead of only providing a URL) affected FAIRness scores. We assumed that a persistent identifier was used when data were made available via data archives that routinely require researchers to provide a minimum of structured information about the survey. We compared FAIRness between those surveys that had a DOI and those that did not. Additionally, we fitted an OLS regression and estimated the effect of an existing DOI on FAIRness scores. To investigate whether DOI availability would impact our findings regarding RQ2, we re-ran our regression models for subsamples: those surveys with a DOI and those without a DOI. We would like to caution that these analyses should not be overinterpreted, in our view, as the availability of a DOI often indicates a decision by survey researchers to publish data via an archive or repository. These archives follow established practices and provide structured metadata that impact the FAIRness score. In this sense, our robustness check serves the purpose of clarifying how much impact this specific decision of data producers for archiving has on the FAIRness score.

Second, we reran our analyses for the full dataset without excluding duplicates. Here, the full sample (N = 717) was used for our analyses. We conducted these analyses for readers who would take the perspective that each survey (wave) should be considered equally, independent of whether it is part of a longitudinal survey program or not. To account for the clustered data structure of the full sample, we re-ran the regression model with cluster-robust standard errors. We further tested a multi-level model but obtained an Intra Class Correlation (ICC) of 0.966, indicating that between survey differences accounted for most of the variation. As including within survey variation would not improve prediction, we discarded this option.

Third, some survey design categories were sparsely populated (e.g., probability-based mixed-mode designs). To avoid overinterpreting unstable estimates, we reran our analyses with separate models for RQ2, including mode and sampling procedure as individual independent variables. To account for the fact that the full sample would yield more cases (which then, however, would be clustered, see above), we additionally reran these models with cluster-robust standard error for the full sample (N = 717).

Fourth, to answer RQ2, we used OLS regressions. As our dependent variables are bounded percentage outcomes, we conducted a robustness check in which we fitted a fractional logit model with overall FAIRness as the dependent variable and the combined survey design variable as the independent variable.

Results

RQ1

Fig 2 shows the distribution of FAIRness among surveys collected during the COVID-19 pandemic. The mean score among the survey was 18.97 out of a maximum of 26 (SD = 4.94), which amounts to 72.96% of FAIRness. The survey that performed the best received 100% of the score, while the worst one received 11.54%. What we observed from the distribution of FAIRness scores is that surveys tend to form two groups. One group of surveys achieved higher scores of around 88.46%, whereas the other achieved scores of about 65.38%.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Distribution of FAIRness of Surveys During the COVID-19 Pandemic.

Note. The figure includes a combination of boxplots with whiskers and outliers as well as onion plots that show the density of the distributions.

https://doi.org/10.1371/journal.pone.0358755.g002

Looking into dimensions of the FAIRness revealed a more nuanced picture. We found that surveys performed best regarding Accessibility with 78.81% on average, followed by Interoperability at 77.74%, Reusability at 71.89%, and, finally, Findability averaging 63.93% of the total score. Again, for each dimension, we observed grouping that differed between dimensions: a majority of surveys were at 85.71% for Accessibility, whereas two groups each can be differentiated for Findability (at 57.14% and 71.43%), Interoperability (at 66.67% and 100.00%), and Reusability (50.00% and 100.00%).

Overall, FAIRness scores varied across the 319 surveys in our data. Here, not every survey received a FAIRness score we would deem desirable. When looking into variation between surveys, especially when focusing on subdimensions, we found grouping that we interpreted as a first indication of systematic differences between surveys. There seems to be a group of surveys that generally achieved a higher level of FAIRness, ensuring the availability and reusability of their data. On the other hand, there seems to be a group of surveys that is not as readily available and accessible for reuse by external researchers.

RQ2

To investigate systematic differences between the FAIRness of surveys in our data, we analyzed FAIRness conditional on their survey design (see Table 1). The use of probability sampling and either CAPI or PAPI mode coincided with the highest average FAIRness of 82.69%, which is closely followed by surveys using probability sampling and CATI mode with FAIRness at 79.81%. The lowest scores in this regard were obtained by surveys using non-probability sampling, regardless of their survey mode.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Summary statistics of FAIRness across different survey designs.

https://doi.org/10.1371/journal.pone.0358755.t001

Our OLS regression confirmed the previous findings (Table 2, Model 1). Our model showed the large differences between probability-based CATI surveys (reference category) and non-probability web surveys (β = −12.268), p <  .001).

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Regression results on the relationship between FAIRness scores and survey design, sampling method, and survey mode.

https://doi.org/10.1371/journal.pone.0358755.t002

Focusing only on the sampling method (Table 2, Model 2) adds nuance to the findings for the combined survey design variable. Surveys using probability sampling scored an average of 78.36% of FAIRness, which was approximately 10 percentage points higher than surveys using non-probability sampling (67.47%). Again, we found this pattern in our regression analyses (β = −10.889), p < .001; Table 2, Model 2).

Similarly, focusing only on survey mode (Table 2, Model 3), surveys that used CAPI obtained the highest FAIRness at 85.9%, whereas those using CAWI mode averaged the lowest at 68.2%. In our regression models (Table 2, Model 3), we observed the same effect (β = −17.697), p < .01).

Overall, our analysis shows an association between survey design and FAIRness of data. Those surveys that utilized designs aimed at fast and cost-efficient production of survey data were also less likely to invest in comprehensive and accessible documentation to make their data easy to reuse.

Conclusion

We set out to investigate the FAIRness of survey data collected in Germany during the COVID-19 pandemic. During this crisis, survey data provided an important foundation to study the population’s attitudes, health-related behaviors, and opinions. Previous studies on data collection practice during the COVID-19 pandemic showed that researchers were facing a trade-off between quickly obtaining data and publishing results versus focusing on implementing high-quality, yet time-consuming, data collection procedures [7–9]. These studies showed that, as a result of this trade-off, surveys that focused on quick data collection and employed the necessary designs for thiswere also less likely to publish data quickly and with comprehensive data documentation. With the present study, we addressed research gaps in this regard. Our data revealed strong variation between the FAIRness scores of surveys (RQ1). We found them to differ systematically, with one group of surveys generally receiving higher scores at around 88.46% of FAIRness, whereas another group of surveys averaged at around 65.38%% FAIRness. This systematic pattern is related to the survey design used during data collection (RQ2). Surveys that employed designs that enabled quick and cost-efficient data collection obtained lower FAIRness scores compared to those with more costly and time-intensive survey designs. In this regard, our findings add further evidence to previous research on data collection practices during the COVID-19 pandemic [6–9] and that the trade-off of collecting data and producing findings quickly also extended to the Findability, Accessibility, Interoperability, and Reusability of these research data.

Our findings have implications for survey researchers, research data management, and data users in general. First, while the need for rapid data collection during crises such as the COVID-19 pandemic is understandable, the FAIRness of data should be considered. Research data are a common good, and enabling replication and transparency are necessary elements of the scientific process. The need to allow replication is especially important when taking on risk regarding the generalizability of findings when operating with survey designs that are at risk of producing lower data quality. Our findings are troublesome news in this regard; exactly those surveys that would require replication to ensure the robustness of their findings obtained the lowest FAIRness scores. Second, academic research is mostly publicly funded. Accordingly, it should be in the interest of funding agencies to make research data publicly available to ensure efficient use of the resources invested and robust results due to replication. As our robustness checks highlighted the importance of publishing data via repositories that require systematic metadata and provide persistent identifiers (i.e., DOIs), we recommend that funding agencies more strongly nudge researchers to publish their research data this way to facilitate the provision of proper metadata.

Our study surely is not without limitations, which future research could address. First, we focused on surveys that were included in the SDCCP data. The SDCCP project did not include every survey, especially not those that were not published anywhere or those that had so little documentation that they were discarded during eligibility testing. In this regard, our study systematically overestimates FAIRness. We therefore caution that real differences and FAIRness scores of surveys in the field will likely be lower.

Second, we relied on automated FAIRness assessment using F-UJI. While this decision enabled us to effectively process documentation for a large number of surveys, the automated procedure has limitations. F-UJI measures require detectable compliance with specific technical patterns and embedded structured metadata. These will be found more likely on DOI-referenced archive-published data compared to project websites. These might even contain rich study documentation, but the algorithm might fail to detect them due to unexpected information structure. For those surveys, for which we only had a URL, we found that FAIRness scores were lower. Future studies could investigate this issue further, for instance, by applying expert review of selected cases to assess possible biases. However, we would argue that the decision to make data available via a data archive or repository that ensures long-term availability as well as systematic and detectable metadata is a decision within the scope of FAIRness itself and, thus, should be considered during such analyses.

Third, we focused on surveys conducted in Germany during the COVID-19 pandemic. Germany is an example of an advanced research landscape with well-developed research infrastructures, for which general knowledge about the need for replicability and transparency in the field should be expected. Focusing on Germany allowed us to draw on an existing collection of metadata on surveys and keep the country context of each survey comparable. However, we would welcome replication in other countries or disciplines. We also see benefits in replicating our analyses without the focus on the COVID-19 pandemic. Analyzing more surveys, however, will require coding the relevant metadata to identify survey design characteristics. The survey selection and coding procedure will require additional resources and time. The procedures developed as part of the SDCCP project, from which we reused data, might serve as a blueprint for such research.

Supporting information