Using a Deliberative Poll on breast cancer screening to assess and improve the decision quality of laypeople

Balancing the benefits and harms of mammography screening is difficult and involves a value judgement. Screening is both a medical and a social intervention, therefore public opinion could be considered when deciding if mammography screening programmes should be implemented and continued. Opinion polls have revealed high levels of public enthusiasm for cancer screening, however, the public tends to overestimate the benefits and underestimate the harms. In the search for better public decision on mammography screening, this study investigated the quality of public opinion arising from a Deliberative Poll. In a Deliberative Poll a representative group of people is brought together to deliberate with each other and with experts based on specific information. Before, during and after the process, the participants’ opinions are assessed. In our Deliberative Poll a representative sample of the Danish population aged between 18 and 70 participated. They studied an online video and took part in five hours of intense online deliberation. We used survey data at four timepoints during the study, from recruitment to one month after the poll, to estimate the quality of decisions by the following outcomes: 1) Knowledge; 2) Ability to form opinions; 3) Opinion stability, and 4) Opinion consistency. The proportion of participants with a high level of knowledge increased from 1% at recruitment to 56% after receiving video information. More people formed an opinion regarding the effectiveness of the screening programme (12%), the economy of the programme (27%), and the ethical dilemmas of screening (10%) due to the process of information and deliberation. For 11 out of 14 opinion items, the within-item correlations between the first two inquiry time points were smaller than the correlations between later timepoints. This indicates increased opinion stability. The correlations between three pairs of opinion items deemed theoretically related a priori all increased, indicating increased opinion consistency. Overall, the combined process of online information and deliberation increased opinion quality about mammography screening by increasing knowledge and the ability to form stable and consistent opinions.


Introduction
Cancer screening is intuitively appealing and is a standard in modern healthcare. It can benefit the population reducing mortality, morbidity and incidence of disease by identifying presymptomatic, localised stage cancer or precursors of cancer [1]. However, screening can also lead to unintended harms [2]. The most profound one, overdiagnosis, occurs in cancer screening when indolent cancers or harmless precursors are identified that were never going to cause harm in a person's remaining lifetime [3]. The harm from overdiagnosis occurs because it is not (yet) possible to distinguish the cancers (and precursors) that do not progress from those that do. They are all, therefore, treated as if they would progress to severe illness. In mammography screening, women treated for a breast cancer (or precursors) that were never going to cause any harm, experience the harms from treatment but none of the benefits [4].
In contrast to most clinical situations, where people respond to a symptom by consulting their doctor, screening is initiated by health authorities and is aimed at the asymptomatic population. Following health authorities' recommendations, it is a fundamental premise that, on average, people are better off participating in screening than not [5]. As the benefits and harms are not measured in the same way, and do not necessarily affect the same people, it is difficult to balance them, and inherent in this process is a value judgement [6]. In addition, screening is not only a medical intervention but also a social intervention posing ethical, legal and social dilemmas and considerations [7]. In this light, it has been argued that the opinions of laypeople should be considered in decisions about screening programmes.
A commonly used way of consulting the public about policy issues is through traditional opinion polls. Findings from such studies reveal widespread enthusiasm with nearly 90% of adult populations in the US and the UK agreeing that "screening is almost always a good idea" [8,9]. In addition, high participation rates in screening programmes could also be argued to reflect support for them. However, both measures require cautious interpretation. Widespread enthusiasm could reflect pseudo-opinions (opinions on a topic about which respondents have no knowledge) [10], or simply top-of-mind opinions based on one-sided information that favours screening [11][12][13], and limited public awareness and understanding of overdiagnosis [14,15]. Despite the increased focus on providing quality information to aid decision making, a study has shown that the decision to attend mammography screening was not based on the information provided in the invitation letter and leaflet [16]. Participation in screening can reflect social norms, perceived moral obligations and responsibilization (the emphasis of responsibility placed on individual women to contribute to their health within society) rather than support for and understanding of the screening programme [17,18].
In order to determine informed laypeople's considered opinions about mammography screening, we conducted a Deliberative Poll 1 [19], originated by Fishkin [20], in Denmark in September 2020. Our Deliberative Poll involved informing a representative sample of people about mammography screening through an online video as well as intense deliberation at an online citizens' assembly. Deliberative Polls combine two normative democratic ideals, representativeness and deliberation, in one method and represent public opinion as it would be if all citizens had had the opportunity to deliberate based on balanced information.
The aim of the study was twofold: first, to design and conduct an online Deliberative Poll with a representative sample of the Danish population engaging in online deliberation. Second, to analyse how information and deliberation affected the participants' knowledge and opinion formation.

Applying citizens' involvement in mammography screening-Communication, assessment of opinions and deliberation
Our Deliberative Poll design on mammography screening had three parts: 1) a video communication about mammography screening; 2) assessment of knowledge and opinions using questionnaires at four timepoints and, 3) a citizens' assembly with a representative sample of the population, neutral moderators, and three experts available to answer questions. The following sections describe the knowledge basis for all three parts as well as the design and outcome hypotheses of the study.
Communication about mammography screening. Informing the public about mammography screening has been subject to intense criticism [21]. Debate concerns which facts should be presented, what constitutes best available evidence, in which format the information should be communicated and if misconceptions should be addressed [22]. Communication about screening is linked to risk and uncertainty: the risk of getting and dying of a disease and the risk of being harmed by the procedures in place to lower the risks. Incorporating personalised risk estimates within communication interventions for screening programmes has been shown to enhance informed choices [23]. Solid risk data about mammography screening is outdated as randomised controlled trials were conducted years ago (mostly in the 1970s and 1980s), with less optimal breast cancer treatment available at the time and risk estimates that are often uncertain because of non-randomised data collection [4]. Risk information is known to be difficult to comprehend. Numerous studies have documented statistical illiteracy among laypeople and physicians [24][25][26]. No standard exists on how to communicate risk most effectively. Metanalyses suggest that numbers should be presented as natural frequencies [27,28], however, the most suitable format depends on the nature of the task [29,30]. Literature suggests that icon arrays may aid understanding and should be formatted consistently with a shared common reference class [29,31]. In addition, recognising uncertainty and presenting screening as a choice are perceived as important by laypeople [32,33]. Taking all of these considerations into account, much effort was put into developing a balanced video about mammography screening for the present study.
Assessing the opinions of laypeople about mammography screening. Opinions about screening, like most social concepts, are complex and multifaceted. The processes involved in opinion formation are not fully understood. Citizens' Juries about mammography screening have provided knowledge about aspects other than harms and benefits that are deemed important to women; for example, the freedom to choose and the symbolic importance of screening -the idea that society cares about women [34,35]. The psychological pathways through which participants process information about overdiagnosis in a mammography screening leaflet, and how this influences their attitudes and intention to participate, have also been studied [36]. The study revealed both cognitive and affective mediators. Knowledge from these studies was used for questionnaire development in the present study, incorporating questions related to the possible cognitive, affective and social aspects of opinions about mammography screening.
The Deliberative Poll-Deliberation and representativeness. To include everyone in a discussion while also ensuring that they are motivated and can reflect on the issue at stake is a difficult task. Nevertheless, the Deliberative Poll is designed to do exactly that [37]. In a Deliberative Poll a representative group of people is brought together to deliberate with each other and with experts based on specific information. Before, during and after the process, the participants' opinions are assessed. In Deliberative Polling, representativeness is defined by compliance with three criteria: 1. Demographics; 2. Attitudes, and 3. Sample size. To support the inference that entire groups of citizens would come to the same conclusions if they had the opportunity to deliberate on the basis of receiving the same information, Deliberative Polls have to comply with demographic and attitudinal representativeness. In addition, a sufficient sample is needed to meaningfully evaluate the representativeness as well as any statistical significance in knowledge and consistency change [37]. It is not straightforward, however, to bring together a representative sample of people to debate a complex issue like mammography screening. Certain groups may self-select or de-select, based on prior knowledge, opinion and engagement with the programme. It is just as difficult to create a free and open atmosphere for deliberation, where everyone can address their concerns and raise questions. As such, the Deliberative Polls' two design-hypotheses can be voiced: (a) it is possible to create representative processes on a national level where (b) deliberation about the issue at stake takes place.
Judging the quality of public opinions-Considerations and hypotheses. From a deliberative democratic standpoint, the quality of opinions arising from a Deliberative Poll can be judged according to the normative democratic ideals related to information and deliberation. If opinions that underlie the decision are informed and deliberate, the decision can be argued to be of good quality. However, information and deliberation have the potential to affect the process of opinion formation in the following three areas-in both desirable and undesirable ways.
First, by affecting knowledge. The detailed information provided to participants along with the opportunity to share knowledge in the deliberative process might bring different types of information into play, thereby increasing the pool of information available to each participant. In addition, the deliberative process allows for clarification of questions. However, as has been documented previously in screening settings, cognitive dissonance might be a barrier to increased knowledge. To be confronted with knowledge that conflicts with previously held beliefs about screening may give rise to feelings of discomfort that can lead to the rejection of new information as a way of coping with the discomfort [38][39][40].
Second, by affecting the ability to form reasoned and stable opinions. The more information and the more arguments the participants encounter during the process of deliberation, the less likely they are to be confronted with unknown arguments and new information later on, which could alter their opinions. Opinion change during the process is expected to occur step by step and not as a random process [41][42][43].
Third, by affecting opinion consistency. The process may lead participants to identify inconsistencies in their own arguments. As inconsistencies make an argument hard to follow in discussions, participants may try to avoid inconsistencies. A counterargument implying that participants become less able to form opinions, and form opinions that are less stable and less consistent, is that complex information and deliberation leave participants more in doubt and more confused.
In light of the normatively desirable and undesirable potentials of information and deliberation already mentioned, we hypothesised as follows: Hypothesis 1: Participants will increase their knowledge about mammography screening as a result of the process of video information and deliberation. Hypothesis 2: The process of information and deliberation will result in an increased ability to form opinions, which will be reflected in fewer 'don't know' answers in the questionnaires and will result in the formation of opinions that are more stable. Hypothesis 3: Participants will have more clarity in their views, which will be reflected in increased consistency between items deemed theoretically related. By testing these hypotheses, this study aims to analyse the quality of public opinion about mammography screening: are participants able to form informed and reasoned opinions on which to base their recommendation?

The Deliberative Poll on mammography screening
The first aim of the study was to conduct an online Deliberative Poll and thereby test the two design hypotheses: (a) it is possible to create a representative process at a national level where (b) deliberation takes place. The following sections describe how this was accomplished in the Deliberative Poll on mammography screening. The Office of Research and Innovation at University of Copenhagen have approved the study, reference 514-0385/19-3000. According to the Committees on Health Research Ethics for the Capital Region of Denmark (De Videnskabsetiske Komiteer for Region Hovedstaden) the project does not constitute a health research project, but is considered a questionnaire-based study as defined by the "Danish Act on Research Ethics Review of Health Research Projects" Section 2. Thus, this project is not subject to notification from the Committees (Journal-no.: 21031705.) The Deliberative Poll was conducted online on Sunday 20 th of September 2020 in Denmark. Using the CJC checklist [44] as a guide, the Deliberative Poll can be described as follows:

Recruitment
Kantar Gallup Denmark ran the recruitment process using their large online panel of 50.000 citizens recruited through random sampled telephone interviews. Citizens were recruited either by their landline or their mobile phone dependent on which number was obtainable. Participants were recruited to the Deliberative Poll through a two-step quota sampling procedure which was closely supervised by MDJ and KMH. First 1,977 men and women from the total Kantar Gallup panel were e-mailed with a link to an online questionnaire (time T 0 ). These participants reflected the sociodemographics of the Danish population aged 18 to 70 by sex, age, educational level, marital status, and electoral district according to Statistics Denmark. Respondents who completed the questionnaire (T 0 ) and agreed to further contact were immediately directed to another online questionnaire (T 1 ). See Fig 1 for the recruitment flow diagram. Answers to T 0 revealed the distribution of responses to key items. Informed consent was obtained written, online in the first questionnaire where participants was provided information about the project. Answering "Yes, I accept" and in addition entering preferred contact information into the questionnaire was considered an informed consent.
The responses to T 0 were used in the second step of quota sampling enabling a sample representative of the following additional parameters: family history of breast cancer, level of worry, education within healthcare, knowledge of and immediate attitude towards mammography screening. Besides enabling representativeness on key items, the second step in the quota sampling also prevented a potential bias arising if, for example, more women than men agreed to participate. A total of 89 people participated in the citizens' assembly. Participant characteristics are shown in Table 1.
The participants were assessed for sociodemographic representativeness by comparing sex, age, education, residence and marital status to the general Danish population aged 18 to 70. These characteristics were derived from 2020 register data from Statistics Denmark. The representativeness of the participants on selected key items (education within healthcare, level of worry, knowledge, initial attitude towards mammography screening and family history of breast cancer) were assessed by comparison to the pre-intervention measures from T 0 . Overall, the target population for the present study was representative of the Danish population of 18 to 70 year-olds ( Table 1). The sampling procedure succeeded in reaching a close to perfect representative sample of the Danish population within the selected age parameters, in line with design hypothesis (a) ( Table 1).

The online citizens' assembly
Participants were invited to take part in what was framed as "a citizen's assembly concerning mammography screening as part of a research project". Inclusion criteria: men and women aged 18-70, who could speak and read Danish. Exclusion criteria: no access to a computer/tablet with camera or microphone. Participants were paid 500 DKK (67 €) in compensation for their time. The assembly was held on a Sunday to minimize selection bias due to participants having to go to work.
The citizens' assembly lasted a day and participants deliberated in small group sessions led by neutral moderators, and in plenary sessions with the opportunity to ask questions of experts. See S1Table for assembly programme. The former chief of the Danish Council on Ethics chaired the assembly and was instructed to take a neutral position. Three experts, looking at the evidence from different perspectives, attended the assembly. One was a medical doctor from the Nordic Cochrane collaboration, one was an anthropologist from the patient organisation, the Danish Cancer Society, and one was a health economist experienced in mammography screening economics. Their role was to answer questions directly from the participants (not to give a general presentation) enabling participants, rather than experts, to define the problems, dilemmas and questions for discussion. The small group discussions (with eight participants in each group) were led by neutral moderators, primarily school teachers, instructed to take a neutral position and to ensure an open and equal dialogue. There was no encouragement or instruction to reach consensus in the groups.
Before the citizens' assembly all participants were encouraged to study a video informing them about mammography screening. The video (in Danish) is available at: http://bit.ly/ mammografi-screening. See S2 Table for a description. In addition to the video, participants were provided with a fact sheet containing key points from the video (see S1 Fig). Sociodemographic characteristics and initial opinions were assessed online at T 0 . Outcomes regarding knowledge, opinions and recommendations were assessed in an online questionnaire at four time points: at the end of recruitment (T 1 ), at the beginning (T 2 ) and end (T 3 ) of the citizen The video was 20 minutes long and was shown at the beginning of the citizens' assembly. Table 2 shows the participants' self-reported compliance with the video intervention as well as participants' ability to complete an item assessing compliance. Compliance was assessed in the citizen's assembly after the screening of the video (T 2 ). 90% of the participants reported compliance in terms of preparation beforehand, as they reported having seen the video twice or more. 85% of participants were able to answer the test question correctly, indicating high compliance with the intervention.
From the participants' evaluation of the group discussions it is evident that they weighted competing arguments. 80% reported that other participants' arguments were useful in the formation of their own opinions. Participants' evaluation of group discussions is described in Table 3. See S3 Table for the full evaluation of the Deliberative Poll and for participants' evaluation of the deliberation.
95% of participants reported that group moderators made sure that all the participants had a chance to be heard (see S3 Table). In addition, 85% of participants reported that the video added new insights on mammography screening (see S3 Table).
The design of the Deliberative Poll together with the participants' evaluation of group moderators and group deliberation support design hypothesis b: an open and free deliberation took place at the citizens' assembly which, together with the video information, provided the basis for learning and opinion formation in the Deliberative Poll.

How information and deliberation affected knowledge and opinion quality in the Deliberative Poll on mammography screening
The second aim of this study was to analyse how the process of information and deliberation affected knowledge and opinion formation. It was hypothesised that: 1) Participants will increase their knowledge about mammography screening as a result of video information and deliberation.
2) The process of information and deliberation will result in an increased ability to form opinions reflected in fewer 'don't know' answers and more stable opinions. 3) Participants will have greater clarity about their views, which will be reflected in increased opinion consistency.

1) Knowledge
Knowledge was conceptualised as correct answers to 13 items addressing conceptual and numeric knowledge expressed with the use of a knowledge index which combined all 13 items, giving 7.69 points for each correct answer. The index ranged from 0 to 100, where 100 indicates correct answers to all questions and 0 indicates incorrect answers to all questions. Low, mid and high levels of knowledge were defined according to the knowledge index as >0, >33.33 and >66.66 respectively. The difference in proportion of high levels of knowledge and the proportion of correct answers to each of the 13 knowledge questions over the four poll inquiry time points (T 1 -T 4 ) were assessed in linear models for binary outcomes adjusting for repeated measurements using generalised estimating equations (GEE). Fig 3 shows the percentage of participants with low, mid and high levels of knowledge. Participants increased their total knowledge markedly (Fig 3). The proportion of participants with a high level of knowledge increased from 1% at recruitment (T 1 ) to 56% after video information (T 2 ). Learning was driven by video information reflected by the increase in knowledge observed between  Table 3. Participants' evaluation of deliberation, T 3 .

Questions T 3 , n = 85%Agreeing
In group discussion we responded to each other's arguments 78 All opinions were listened to with the same respect 89 I got an understanding of other participants' arguments that contradicted my own 67 Other participants' arguments were useful in the formation of my own opinions 80 Note: T 3 = inquiry time point 3 (after deliberation). https://doi.org/10.1371/journal.pone.0258869.t003

PLOS ONE
T 1 and T 2 . There was no statistically significant difference in knowledge before and after the citizens' assembly, T 2 vs T 3 . A decrease in knowledge is seen one month after the assembly (T 4 ), but most of the gained knowledge is persistent.

PLOS ONE
The development of knowledge expressed as a 0-100 index over the four time points, stratified for sociodemographic participant characteristics, was assessed in a linear model adjusting for repeated measurements using GEE. Learning was not restricted to specific participants as learning is seen across almost all subgroups (S4 Table). We found the largest knowledge gain in participants with the lowest level of knowledge at recruitment (T 1 ), but also noticeable knowledge gain in participants with middle levels of knowledge (S4 Table). It is impossible to conclude anything statistically significant about learning in the group of people with high levels of knowledge at recruitment (T 1 ) due to ceiling effect and because only one participant belonged to this group. Knowledge gain was also investigated for each of the 13 knowledge items separately in linear models for binary (correct vs incorrect) outcomes accounting for repeated measurements using GEE. The combined process of information and deliberation was followed by a statistically significant increase in knowledge on most items (11 out of 13, S5 Table).
To assess the occurrence of selective learning, participants were divided according to their initial recommendation about mammography screening. Participants were also divided according to decisiveness-undecided participants answering 'don't know' vs. decided participants giving any other answer. The differences in development in the knowledge index and the proportion of correct answers to each of the 13 knowledge questions over the four inquiry time points between these groups were assessed in linear models using GEE. There was no indication of selective learning (Table 4).: there was no statistically significant difference in learning between the participants with different initial opinions ("continue" vs "expand"). As the initial opinion of "discontinue" was only shared by two participants no statistical analysis was conducted comparing this group to the others. On two knowledge items there was a statistically significant difference in learning between the decided and undecided, pointing, however, in opposite directions, which may be a spurious result due to multiple testing.
These results support our first hypothesis: participants increased their knowledge about mammography screening.

2) Ability to form opinions and opinion stability
Participants' ability to form opinions was conceptualised as their ability to answer opinion questions with anything other than 'don't know'. The ability to form opinions for each of the 14 items over the four poll inquiry time points was assessed in linear models for binary outcomes using GEE. Statistically significantly more people formed an opinion regarding effect size (12%), costs (27%) and ethical dilemmas in screening (10%), as a drop in 'don't know' answers was seen comparing responses at recruitment (T 1 ) to responses given after deliberation (T 3 ) (Fig 4 and S6 Table). These results support our second hypothesis: the process of information and deliberation increased participants' ability to form opinions.
Question wording related to the item "Balance": Below are two statements. You can think of them as a discussion between two people, A and B. We will ask you to answer if you are most in agreement with A or most in agreement with B. Even if you do not totally agree with any of the positions, we will ask you to answer which position comes closest to your own viewpoint. How do you consider the balance between benefits and harms in mammography screening? A says: The harm carries most weight. Too many women are overdiagnosed and overtreated and getting false alarms compared to the women prevented from dying of breast cancer. B says: The benefit carries most weight. The number of women prevented from dying of breast cancer counterbalances the number of women who are overdiagnosed and overtreated and those getting false alarms. T 1 = inquiry time point 1 (recruitment), T 2 = inquiry time point 2 (after video information), T 3 = inquiry time point 3 (after deliberation), T 4 = inquiry time point 4 (one month after the citizens' assembly).
Stability was operationalised as correlations within opinion items at different time points. For 11 out of 14 opinion items, the within item correlations between the time of recruitment (T 1 ) and the time after video information (T 2 ) were smaller than the correlations between the time after video information (T 2 ) and the time after deliberation (T 3 ) (S3 Fig), indicating increased opinion stability as a result of the process. In addition, for all items the mean of sequential order correlations was larger than the mean of non-sequential order correlations (Fig 5) indicating that the opinion change during the deliberative polling did not reflect a Table 4

Learning from T 1 to T 3 Recommendation about mammography screening in Denmark (T 1 )
Decisive regarding recommendation (T 1 ); Discontinue continue, and expand put together. n = 84 random process. These results support our second hypothesis: the process of information and deliberation lead to more stable opinions.

Opinion consistency
Consistency was operationalised as correlations between the 14 opinion items within each timepoint based on Converse's ideas [42]. Even though correlations are not measuring consistency but covariance, they are used as a measure for consistency as they specify whether variation in one variable corresponds to variation in another [45].

PLOS ONE
In all correlation analyses 'don't know' answers were excluded. The number of statistically significant between-item correlations increased from 40 at recruitment (T 1 ) to 46 after the process of information and deliberation (T 3 ) (Figs 6 and S4). Of the 33 between-item correlations that were statistically significant at both T 1 and T 3 , 22 increased from T 1 to T 3 , ten decreased and one remained the same. Three between-item correlations deemed theoretically related a priori increased from T 1 to T 3 : Support-Balance (from -0.28 to -0.55); Support-Effect (from -0.37 to -0.44), and Support-Costs (from 0.31 to 0.49). These results support our third hypothesis: the process of information and deliberation increased consistency between items deemed theoretically related, reflecting that participants had more clarity about their views.

Discussion
To our knowledge we conducted the first Deliberative Poll on mammography screening and therefore our study is the first to report on laypeople's decision quality regarding mammography screening using this method. The method however have been widely used within other areas [46,47] and assessment of opinion quality using this method have been done before [43].
Our study revealed that it is possible for laypeople to form considered and informed opinions about mammography screening based on a process of video information and online deliberation. It might be supposed that citizens could be in greater doubt about their opinions when engaging on such a complex issue, but we found that more people formed opinions after information and deliberation compared to before. At the moment, the public is not in a position to form reasoned opinions about screening as baseline knowledge is low, which has been shown in several other studies [48][49][50]. In line with our expectations, laypeople were able to gain complex knowledge about mammography screening. Conceptual knowledge increased relatively more than numerical knowledge, even though the definition of numerical knowledge permitted a relatively wide range of correct answers (S5 Table). Despite efforts to aid numerical learning (rounded numbers and provision of a fact sheet with numbers), numerical

PLOS ONE
knowledge gain was relatively lower (21-33%) after information and deliberation (S5 Table), but still substantially increased compared to time of recruitment. Conceptual knowledge, on the other hand, was high after information and deliberation where more than 80% gave correct responses to seven out of nine items.
In line with our expectations, more people formed opinions during the deliberative process as more took a stand on screening effect size, costs and ethical dilemmas in screening (Fig 4,  S6 Table). Moreover, and surprisingly, people in general expressed a formed opinion on most items at recruitment (T 1 ), as few answered 'don't know' to the questions. It could simply be that people already do have views on the topic. An alternative interpretation could be that high levels of media attention towards preventive strategies generate 'top-of-mind' opinions, or that people do not like to admit that they 'don't know'. The latter has been demonstrated in studies, where people respond with apparent opinions on named legislation that does not exist [10]. However, the questionnaires in the present study began with text addressing the possibility of answering 'don't know' as an attempt to comply with the described phenomenon. As no increase in 'don't know' responses was seen, participants did not become more doubtful of their opinions.
Participants, among whom the majority favoured screening at recruitment, weighted competing arguments during deliberation (Tables 4 and S3), and 89% said that all opinions were listened to with the same respect while 67% said that they gained an understanding of other participants' arguments that contradicted their own. Opinion stability increased (Figs 5 and S3) and consistency between theoretically related opinion items also increased (Figs 6 and S4) indicating opinion change related to the deliberative process and not a random change. 33% of participants felt there was too little time to discuss in group sessions whereas 34% felt there was enough time (S3 Table). More time for deliberation might have helped participants in doubt (answering 'don't know').
Participants evaluated the video positively: 71% did not feel that the video pushed their opinion in a certain direction whereas 29% felt that it did (S3 Table). We do not know how many of those 29% viewed the video as slanting away from versus slanting towards mammography screening. Looking at findings from a randomised controlled trial comparing two decision aids about mammography screening, one that contained information on overdetection and one that did not, it could be both [12]. It might be that when a population already enthusiastic about screening is informed about harms, the new information that conflicts with their main beliefs is viewed as pulling opinions in the opposite direction. Overall, participants in the Deliberative Poll evaluated the video information very positively: 96% reported that the video added new insights on mammography screening; 76% reported that it made their opinions more nuanced; 80% felt prepared to form opinions about mammography screening based on the video, and 69% would recommend it to peers (S3 Table).
The video had the greatest influence on participants' knowledge as seen in Fig 3. However, the majority of participants (68%) reported plenary discussions with experts as having the greatest impact on their opinions. The combination of video information and deliberation at the citizens' assembly seems to complement each other well. Overall, the study demonstrated that it is possible to gather a representative group of people in meaningful online deliberation about mammography screening, and 85% of the study participants reported that they would recommend participating in a citizens' assembly to peers.
We acknowledge that our results should be viewed in light of the evidence we provided to participants, how we conceptualised decision quality as well as our deliberative poll setup including time for deliberation.

Strength and limitations
An advantage of this study was the rigorous way the design ensured close to perfect sociodemographic as well as attitudinal representativeness. Another advantage relates to the repeated measurement design. This longitudinal design with four waves of items addressed to the same participants ensured statistical power without an even larger sample size. In addition, the design allowed for analysis over time. Both women (with and without a history of breast cancer) and men were included in the Deliberative Poll. This was to gain insights into the general public's perspectives about mammography screening in the light of the involved trade-offs when providing a healthcare service. This is in contrast to most citizens' juries which tend to include only women without a history of breast cancer [33][34][35]51]. Compared to a physical assembly, the online format brought cost savings related to travel, venue and catering. Our findings should be interpreted in light of two limitations.
First, selection bias. The initial response rate was 63%. As with all studies where participation is not mandatory, some selection bias is inevitable. Attempts made to minimize selection bias were the provision of an online rehearsal with the purpose of familiarising the participants with the online platform and their technical equipment at home. Around 40% took part in one of the two rehearsals held in the days before the assembly. The two-step quota sampling process meant that the response rate below 100% did not affect representativeness on the known sociodemographic and attitudinal parameters.
Second, retest bias. We cannot exclude the possibility that repeated measurement in this study in itself leads to an increase in performance on knowledge items [52]. An equivalent phenomenon related to opinion formation is the Socratic effect-where questions in themselves can cause the respondent to reflect on their opinions and behaviour, leading to increased consistency [41,53]. Inclusion of a control group tested only after viewing the video could have been beneficial to assess any potential retest effect. However, as the impact of the study intervention on knowledge was large, we believe that most of the increased knowledge resulted from actual learning and not a retest phenomenon.
Our study shows that citizens' engagement through Deliberative Polling improves their knowledge, opinion consistency, stability and decision-making ability on the very complex issue of balancing benefits and harms in mammography screening.

Implications
This study provides much optimism around how to engage with public opinion in the future concerning complex health programmes. To our knowledge, this is the first study to demonstrate that it is possible to involve a representative group of people at a national level in online deliberation. In light of the ongoing COVID-19 pandemic, this is an important finding as physical assemblies are currently prohibited in many countries and may remain an area of uncertainty in the years to come. The online format is also cost and time saving, and environmentally more sustainable, giving it great potential in future deliberative events. If there is a need for engagement of the public, using an online Deliberative Poll is both feasible and cheap.
It is the politicians who decide if a screening programme should be implemented based on recommendations made by experts. Recommendations from the deliberative poll could be used to inform this policy. Engaging with the public have the potential to increase the chance of a successful implementation or termination of screening programmes due to increased understanding of the different positions and reasons underlying the decision.